support large folio swap-out and swap-in for shmem

Add swap available space validation before spliting shmem large folio to avoid redundant split, since we can not write shmem folio to the swap device in this case.

Signed-off-by Baolin Wang <[email protected]>

---
 mm/vmscan.c | 8 ++++++++
 1 file changed, 8 insertions(+)

mm/vmscan.c +8 -1

index 1807e5d95dda..61465f92283f 100644
@@ -1237,6 +1237,14 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
                        }
                } else if (folio_test_swapbacked(folio) &&
                           folio_test_large(folio)) {
+
+                       /*
+                        * Do not split shmem folio if no swap memory
+                        * available.
+                        */
+                       if (!total_swap_pages)
+                               goto activate_locked;
+
                        /* Split shmem folio */
                        if (split_folio_to_list(folio, folio_list))
                                goto keep_locked;
--

2.39.3

Quoted 4 lines · Baolin Wang

Add swap available space validation before spliting shmem large folio to avoid redundant split, since we can not write shmem folio to the swap device in this case.

We don't scan anon LRU unless we know we may be able to swap. Even if there is swap space, we may still not be able to swap. See can_reclaim_anon_pages().

Quoted 28 lines

Signed-off-by Baolin Wang <[email protected]>

---
  mm/vmscan.c | 8 ++++++++
  1 file changed, 8 insertions(+)

mm/vmscan.c +8 -1

index 1807e5d95dda..61465f92283f 100644
@@ -1237,6 +1237,14 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
                         }
                 } else if (folio_test_swapbacked(folio) &&
                            folio_test_large(folio)) {
+
+                       /*
+                        * Do not split shmem folio if no swap memory
+                        * available.
+                        */
+                       if (!total_swap_pages)
+                               goto activate_locked;
+
                         /* Split shmem folio */
                         if (split_folio_to_list(folio, folio_list))
                                 goto keep_locked;
--

2.39.3

Shmem will support large folio allocation [1] [2] to get a better performance, however, the memory reclaim still splits the precious large folios when trying to swap-out shmem, which may lead to the memory fragmentation issue and can not take advantage of the large folio for shmeme.

Moreover, the swap code already supports for swapping out large folio without split, and large folio swap-in[3] series is queued into mm-unstable branch. Hence this patch set also supports the large folio swap-out and swap-in for shmem.

Functional testing
==================
I use the latest mm-unstable branch to test with reverting Chris's "mm: swap: mTHP swap allocator base on swap cluster order" series which can cause some problems (Hugh also reported in [4]).

Quoted 5 lines · On Tue, 18 Jun 2024 14:54:12 +0800 Baolin Wang <[email protected]>…

Shmem will support large folio allocation [1] [2] to get a better performance, however, the memory reclaim still splits the precious large folios when trying to swap-out shmem, which may lead to the memory fragmentation issue and can not take advantage of the large folio for shmeme.

Quoted 4 lines

Moreover, the swap code already supports for swapping out large folio without split, and large folio swap-in[3] series is queued into mm-unstable branch. Hence this patch set also supports the large folio swap-out and swap-in for shmem.

I'll add this to mm-unstable for some exposure, but I wonder how much testing it will have recieved by the time the next merge window opens?

Quoted 17 lines · Andrew Morton

Quoted 13 lines · On Tue, 18 Jun 2024 14:54:12 +0800 Baolin Wang <[email protected]>…

Shmem will support large folio allocation [1] [2] to get a better performance,
however, the memory reclaim still splits the precious large folios when trying
to swap-out shmem, which may lead to the memory fragmentation issue and can not
take advantage of the large folio for shmeme.

Moreover, the swap code already supports for swapping out large folio without
split, and large folio swap-in[3] series is queued into mm-unstable branch. Hence this patch set also supports the large folio swap-out and swap-in for shmem.

Quoted 2 lines

I'll add this to mm-unstable for some exposure, but I wonder how much testing it will have recieved by the time the next merge window opens?

Thanks Andrew. I am fine with this series going to 6.12 if you are concerned about insufficient testing (and let's also wait for Hugh's comments). Since we (Daniel and I) have some follow-up patches that will rely on this swap series, hope this series can be tested as extensively as possible to ensure its stability in the mm branch.

Quoted 18 lines · Baolin Wang

Quoted 17 lines · Andrew Morton

Quoted 13 lines · On Tue, 18 Jun 2024 14:54:12 +0800 Baolin Wang <[email protected]>…

Shmem will support large folio allocation [1] [2] to get a better performance,
however, the memory reclaim still splits the precious large folios when trying
to swap-out shmem, which may lead to the memory fragmentation issue and can not
take advantage of the large folio for shmeme.

Moreover, the swap code already supports for swapping out large folio without
split, and large folio swap-in[3] series is queued into mm-unstable branch. Hence this patch set also supports the large folio swap-out and swap-in for shmem.

Quoted 8 lines

Quoted 2 lines

I'll add this to mm-unstable for some exposure, but I wonder how much testing it will have recieved by the time the next merge window opens?

Thanks Andrew. I am fine with this series going to 6.12 if you are concerned about insufficient testing (and let's also wait for Hugh's comments). Since we (Daniel and I) have some follow-up patches that will rely on this swap series, hope this series can be tested as extensively as possible to ensure its stability in the mm branch.

Thanks for giving it the exposure, Andrew, but please drop it from mm-unstable until the next cycle. I'd been about to write to say I wouldn't be trying it until next cycle, when your mm-commits came in:
so I thought I ought at least to give mm-everything-2024-06-18 a try.

Baolin may have fixed stuff, but he (or the interaction with other mm work) has broken stuff too: I couldn't get as far with it as with the previous version. Just "cp -a" of a kernel source tree into a tmpfs huge=always size=<bigenough> failed with lots of ENOSPCs, and when "rm -rf"ed lots of WARN_ON(inode->i_blocks) from shmem_evict_inode(); and on second attempt, then a VM_BUG_ON_FOLIO(!folio_contains) from find_lock_entries().

Or maybe that VM_BUG_ON_FOLIO() was unrelated, but a symptom of the bug

I'm trying to chase even when this series is reverted: some kind of page double usage, manifesting as miscellaneous "Bad page"s and VM_BUG_ONs, mostly from page reclaim or from exit_mmap(). I'm still getting a feel for it, maybe it occurs soon enough for a reliable bisection, maybe not.

(While writing, a run with mm-unstable cut off at 2a9964cc5d27, drop KSM_KMEM_CACHE(), instead of reverting just Baolin's latest, has not yet hit any problem: too early to tell but promising.)

And before 2024-06-18, I was working on mm-everything-2024-06-15 minus Chris Li's mTHP swap series: which worked fairly well, until it locked up with __try_to_reclaim_swap()'s filemap_get_folio() spinning around on a page with 0 refcount, while a page table lock is held which one by one the other CPUs come to want for reclaim. On two machines.

None of these problems seen on Stephen's last next-2024-06-13. I had wanted to see if mm-everything-2024-06-18 fixed that lockup, but with the new problems I cannot tell (or it could all be the same
problem: but if so, odd that it manifests consistently differently).

There are way too many mTHP shmem and swap patch series floating around at the moment, in mm and in fs, for me to cope with:
everyone, please slow down and test more.

Quoted 21 lines · Baolin Wang

Quoted 19 lines · Andrew Morton

Quoted 8 lines · On Tue, 18 Jun 2024 14:54:12 +0800 Baolin Wang <[email protected]>…

Shmem will support large folio allocation [1] [2] to get a better performance,
however, the memory reclaim still splits the precious large folios when trying
to swap-out shmem, which may lead to the memory fragmentation issue and can not
take advantage of the large folio for shmeme.

Quoted 5 lines

Moreover, the swap code already supports for swapping out large folio without
split, and large folio swap-in[3] series is queued into mm-unstable branch. Hence this patch set also supports the large folio swap-out and swap-in for shmem.

I'll add this to mm-unstable for some exposure, but I wonder how much testing it will have recieved by the time the next merge window opens?

Quoted 5 lines

Thanks Andrew. I am fine with this series going to 6.12 if you are concerned about insufficient testing (and let's also wait for Hugh's comments). Since we (Daniel and I) have some follow-up patches that will rely on this swap series, hope this series can be tested as extensively as possible to ensure its stability in the mm branch.

Thanks for giving it the exposure, Andrew, but please drop it from mm-unstable until the next cycle.

p.s. I think Andrew Bresticker's do_set_pmd() fix has soaked long enough, and deserves promotion to hotfix and Linus soon.

Quoted 28 lines · On Wed, 19 Jun 2024 01:16:42 -0700 (PDT) Hugh Dickins <[email protected]> wrote:

Quoted 26 lines · Baolin Wang

Quoted 11 lines · Andrew Morton

Quoted 8 lines · On Tue, 18 Jun 2024 14:54:12 +0800 Baolin Wang <[email protected]>…

Shmem will support large folio allocation [1] [2] to get a better performance,
however, the memory reclaim still splits the precious large folios when trying
to swap-out shmem, which may lead to the memory fragmentation issue and can not
take advantage of the large folio for shmeme.

Quoted 8 lines

Quoted 5 lines

Moreover, the swap code already supports for swapping out large folio without
split, and large folio swap-in[3] series is queued into mm-unstable branch. Hence this patch set also supports the large folio swap-out and swap-in for shmem.

I'll add this to mm-unstable for some exposure, but I wonder how much testing it will have recieved by the time the next merge window opens?

Thanks Andrew. I am fine with this series going to 6.12 if you are concerned about insufficient testing (and let's also wait for Hugh's comments). Since we (Daniel and I) have some follow-up patches that will rely on this swap series, hope this series can be tested as extensively as possible to ensure its stability in the mm branch.

Quoted 2 lines

Thanks for giving it the exposure, Andrew, but please drop it from mm-unstable until the next cycle.

Thanks, dropped.

Quoted 3 lines

p.s. I think Andrew Bresticker's do_set_pmd() fix has soaked long enough, and deserves promotion to hotfix and Linus soon.

Oh, OK, done.

And it's cc:stable. I didn't get any sens of urgency for this one - what is your thinking here?

Quoted 29 lines · Andrew Morton

Quoted 28 lines · On Wed, 19 Jun 2024 01:16:42 -0700 (PDT) Hugh Dickins <[email protected]> wrote:

Quoted 26 lines · Baolin Wang

Quoted 11 lines · Andrew Morton

Quoted 8 lines · On Tue, 18 Jun 2024 14:54:12 +0800 Baolin Wang <[email protected]>…

Shmem will support large folio allocation [1] [2] to get a better performance,
however, the memory reclaim still splits the precious large folios when trying
to swap-out shmem, which may lead to the memory fragmentation issue and can not
take advantage of the large folio for shmeme.

Quoted 8 lines

Quoted 5 lines

Moreover, the swap code already supports for swapping out large folio without
split, and large folio swap-in[3] series is queued into mm-unstable branch. Hence this patch set also supports the large folio swap-out and swap-in for shmem.

I'll add this to mm-unstable for some exposure, but I wonder how much testing it will have recieved by the time the next merge window opens?

Thanks Andrew. I am fine with this series going to 6.12 if you are concerned about insufficient testing (and let's also wait for Hugh's comments). Since we (Daniel and I) have some follow-up patches that will rely on this swap series, hope this series can be tested as extensively as possible to ensure its stability in the mm branch.

Quoted 4 lines

Quoted 2 lines

Thanks for giving it the exposure, Andrew, but please drop it from mm-unstable until the next cycle.

Thanks, dropped.

Thanks. I'll add a little more info in other mail, against the further 2024-06-18 problems I reported, but tl;dr is they are still a mystery:
I cannot yet say "drop this" or "drop that" or "here's a fix".

Quoted 8 lines

Quoted 3 lines

p.s. I think Andrew Bresticker's do_set_pmd() fix has soaked long enough, and deserves promotion to hotfix and Linus soon.

Oh, OK, done.

And it's cc:stable. I didn't get any sens of urgency for this one - what is your thinking here?

I thought you were right to add the cc:stable. The current v6.8..v6.10 state does not result in any crashes or warnings, but it totally (well, 511 times out of 512, in some workloads anyway) defeats the purpose of shmem+file huge pages - the kernel is going to all the trouble of trying to allocate those huge pages, but then refuses to map them by PMD unless the fault happens to occur within the first 4096 bytes (talking x86_64).

I imagine that this causes a significant performance degradation in some workloads which ask for and are used to getting huge pages there; and they might also exceed their memory quotas, since a page table has to be allocated where a PMD-mapping needs none (anon THPs reserve a page table anyway, to rely on later if splitting, but shmem+file THPs do not). And it's surprising that no tests were reported as failing.

and on second attempt, then a VM_BUG_ON_FOLIO(!folio_contains) from find_lock_entries().

Or maybe that VM_BUG_ON_FOLIO() was unrelated, but a symptom of the bug I'm trying to chase even when this series is reverted:

some kind of page
double usage, manifesting as miscellaneous "Bad page"s and VM_BUG_ONs, mostly from page reclaim or from exit_mmap(). I'm still getting a feel for it, maybe it occurs soon enough for a reliable bisection, maybe not.

(While writing, a run with mm-unstable cut off at 2a9964cc5d27, drop KSM_KMEM_CACHE(), instead of reverting just Baolin's latest, has not yet hit any problem: too early to tell but promising.)

And before 2024-06-18, I was working on mm-everything-2024-06-15 minus Chris Li's mTHP swap series: which worked fairly well, until it locked up with __try_to_reclaim_swap()'s filemap_get_folio() spinning around on a page with 0 refcount, while a page table lock is held which one by one the other CPUs come to want for reclaim. On two machines.

(I do have doubts about Barry's: the "_new" in folio_add_new_anon_rmap() was all about optimizing a known-exclusive case, so it surprises me to see it being extended to non-exclusive; and I worry over how its atomic_set(&page->_mapcount, 0)s can be safe when non-exclusive (but I've never caught up with David's exclusive changes, I'm out of date).

But even if those are wrong, I'd expect them to tend towards a mapped page becoming unreclaimable, then "Bad page map" when munmapped, not to any of the double-free symptoms I've actually seen.)

Quoted 6 lines

(I do have doubts about Barry's: the "_new" in folio_add_new_anon_rmap() was all about optimizing a known-exclusive case, so it surprises me to see it being extended to non-exclusive; and I worry over how its atomic_set(&page->_mapcount, 0)s can be safe when non-exclusive (but I've never caught up with David's exclusive changes, I'm out of date).

We discussed that a while ago: if we wouldn't be holding the folio lock in the "folio == swapcache" at that point (which we do for both do_swap_page() and unuse_pte()) things would already be pretty broken.

That's I added a while ago:

if (unlikely(!folio_test_anon(folio))) {
	VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio);
	/*
	 * For a PTE-mapped large folio, we only know that the single
	 * PTE is exclusive. Further, __folio_set_anon() might not get
	 * folio->index right when not given the address of the head
	 * page.
	 */
	...

We should probably move that VM_WARN_ON_FOLIO() to folio_add_new_anon_rmap() and document that it's required in the non-exclusive case.

Quoted 1 line

Quoted 3 lines

But even if those are wrong, I'd expect them to tend towards a mapped page becoming unreclaimable, then "Bad page map" when munmapped, not to any of the double-free symptoms I've actually seen.)

What's the first known-good commit?

Quoted 10 lines · David Hildenbrand

Quoted 6 lines

(I do have doubts about Barry's: the "_new" in folio_add_new_anon_rmap() was all about optimizing a known-exclusive case, so it surprises me to see it being extended to non-exclusive; and I worry over how its atomic_set(&page->_mapcount, 0)s can be safe when non-exclusive (but I've never caught up with David's exclusive changes, I'm out of date).

We discussed that a while ago: if we wouldn't be holding the folio lock in the "folio == swapcache" at that point (which we do for both do_swap_page() and unuse_pte()) things would already be pretty broken.

You're thinking of the non-atomic-ness of atomic_set(): I agree that the folio lock makes that safe (against other adds; but not against concurrent removes, which could never occur with the old "_new" usage).

But what I'm worried about is the 0 in atomic_set(&page->_mapcount, 0):
once the folio lock has been dropped, another non-exclusive add could come in and set _mapcount again to 0 instead of to 1 (mapcount 1 when it should be 2)?

Quoted 18 lines

That's I added a while ago:

if (unlikely(!folio_test_anon(folio))) {
	VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio);
	/*
	 * For a PTE-mapped large folio, we only know that the single
	 * PTE is exclusive. Further, __folio_set_anon() might not get
	 * folio->index right when not given the address of the head
	 * page.
	 */
	...

We should probably move that VM_WARN_ON_FOLIO() to folio_add_new_anon_rmap() and document that it's required in the non-exclusive case.

Quoted 1 line

Quoted 5 lines

Quoted 3 lines

But even if those are wrong, I'd expect them to tend towards a mapped page becoming unreclaimable, then "Bad page map" when munmapped, not to any of the double-free symptoms I've actually seen.)

What's the first known-good commit?

I cannot answer that with any certainty: we're on the shifting sands of next and mm-unstable, and the bug itself is looking rather like something which gets amplified or masked by other changes - witness my confident arrival at Barry's series as introducing the badness, only for a longer run then to contradict that conclusion.

There was no sign of a problem in a 20-hour run of the same load on rc3-based next-2024-06-13 (plus my posted fixes); there has been no sign of this problem on 6.10-rc1, rc2, rc3 (but I've not tried rc4 itself, mm.git needing the more urgent attention). mm-everything- 2024-06-15 (minus Chris's mTHP swap) did not show this problem, but did show filemap_get_folio() hang. mm-everything-2024-06-18 (minus Baolin's mTHP shmem swap) is where I'm hunting it.

Quoted 6 lines · Hugh Dickins

and on second attempt, then a VM_BUG_ON_FOLIO(!folio_contains) from find_lock_entries().

Or maybe that VM_BUG_ON_FOLIO() was unrelated, but a symptom of the bug I'm trying to chase even when this series is reverted:

Yes, I doubt now that the VM_BUG_ON_FOLIO(!folio_contains) was related to Baolin's series: much more likely to be an instance of other problems.

Quoted 8 lines

some kind of page
double usage, manifesting as miscellaneous "Bad page"s and VM_BUG_ONs, mostly from page reclaim or from exit_mmap(). I'm still getting a feel for it, maybe it occurs soon enough for a reliable bisection, maybe not.

(While writing, a run with mm-unstable cut off at 2a9964cc5d27, drop KSM_KMEM_CACHE(), instead of reverting just Baolin's latest, has not yet hit any problem: too early to tell but promising.)

Yes, that ran without trouble for many hours on two machines. I didn't do a formal bisection, but did appear to narrow it down convincingly to Barry's folio_add_new_anon_rmap() series: crashes soon on both machines with Barry's in but Baolin's out, no crashes with both out.

Yet while I was studying Barry's patches trying to explain it, one of the machines did at last crash: it's as if Barry's has opened a window which makes these crashes more likely, but not itself to blame.

I'll go back to studying that crash now: two CPUs crashed about the same time, perhaps they interacted and give a hint at root cause.

(I do have doubts about Barry's: the "_new" in folio_add_new_anon_rmap() was all about optimizing a known-exclusive case, so it surprises me to see it being extended to non-exclusive; and I worry over how its atomic_set(&page->_mapcount, 0)s can be safe when non-exclusive (but I've never caught up with David's exclusive changes, I'm out of date).

But even if those are wrong, I'd expect them to tend towards a mapped page becoming unreclaimable, then "Bad page map" when munmapped, not to any of the double-free symptoms I've actually seen.)

Quoted 1 line

Quoted 5 lines

And before 2024-06-18, I was working on mm-everything-2024-06-15 minus Chris Li's mTHP swap series: which worked fairly well, until it locked up with __try_to_reclaim_swap()'s filemap_get_folio() spinning around on a page with 0 refcount, while a page table lock is held which one by one the other CPUs come to want for reclaim. On two machines.

I've not seen that symptom at all since 2024-06-15: intriguing, but none of us can afford the time to worry about vanished bugs.