mm/page_io: fix PSWPIN undercount for large folios in sio_read_complete()
Subsystems (2)
PATCH v2 1/1 mm/page_io: fix PSWPIN undercount for large folios in sio_read_complete()
sio_read_complete() uses sio->pages to account global PSWPIN vm events, but sio->pages tracks the number of bvec entries (folios), not base pages. For large folios this undercounts compared to the per-memcg path which correctly uses folio_nr_pages(), and compared to the bdev read paths which also use folio_nr_pages().
Use sio->len >> PAGE_SHIFT instead, which gives the correct base page count since sio->len is accumulated via folio_size(folio).
--- mm/page_io.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/page_io.c +1 -1
@@ -497,7 +497,7 @@ static void sio_read_complete(struct kiocb *iocb, long ret)folio_mark_uptodate (folio);folio_unlock (folio); } -count_vm_events (PSWPIN , sio->pages); +count_vm_events (PSWPIN , sio->len >>PAGE_SHIFT ); }else {for (p =0 ; p < sio->pages; p++) {struct folio *folio =page_folio (sio->bvec[p].bv_page);
Signature · 1 line
2.53.0
Andrew Morton
maintainerQuoted 8 lines · On Mon, 30 Mar 2026 08:12:29 +0100 David Carlier <[email protected]> wrote:
sio_read_complete()usessio->pagesto account global PSWPIN vm events, butsio->pagestracks the number of bvec entries (folios), not base pages. For large folios this undercounts compared to the per-memcg path which correctly usesfolio_nr_pages(), and compared to the bdev read paths which also usefolio_nr_pages().Use
sio->len>>PAGE_SHIFTinstead, which gives the correct base page count sincesio->lenis accumulated viafolio_size(folio).
Thanks. What are the userspace-visible runtime effects of the bug? Just /proc/vmstat wrongness?
Seems pretty minor, and it's been in there for several years. But I guess I'll slip a cc:stable in there, let it trickle back at a later time.
Quoted 14 lines
mm/page_io.c +1
-1@@ -497,7 +497,7 @@ static void sio_read_complete(struct kiocb *iocb, long ret)folio_mark_uptodate (folio);folio_unlock (folio); } -count_vm_events (PSWPIN , sio->pages); +count_vm_events (PSWPIN , sio->len >>PAGE_SHIFT ); }else {for (p =0 ; p < sio->pages; p++) {struct folio *folio =page_folio (sio->bvec[p].bv_page);Signature · 1 line
2.53.0
Quoted 7 lines · David Carlier
sio_read_complete()usessio->pagesto account global PSWPIN vm events, butsio->pagestracks the number of bvec entries (folios), not base pages. For large folios this undercounts compared to the per-memcg path which correctly usesfolio_nr_pages(), and compared to the bdev read paths which also usefolio_nr_pages().
Quoted 5 lines
Use
sio->len>>PAGE_SHIFTinstead, which gives the correct base page count sincesio->lenis accumulated viafolio_size(folio).Fixes a1a0dfd56f97 ("mm: handle THP in swap_*page_fs()")Signed-off-by David Carlier <[email protected]>
The patch seems theoretically correct, but I’m wondering where we can swap in mTHP for filesystem-based swap?
In both do_swap_page() and shmem_swapin_folio(), we check data_race(si->flags & SWP_SYNCHRONOUS_IO) before allocating large folios. Am I missing something?
Quoted 18 lines
--- mm/page_io.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/page_io.c +1
-2@@ -497,7 +497,7 @@ static void sio_read_complete(struct kiocb *iocb, long ret)folio_mark_uptodate (folio);folio_unlock (folio); } -count_vm_events (PSWPIN , sio->pages); +count_vm_events (PSWPIN , sio->len >>PAGE_SHIFT ); }else {for (p =0 ; p < sio->pages; p++) {struct folio *folio =page_folio (sio->bvec[p].bv_page); --
Quoted 2 lines
2.53.0
Thanks
Barry
Quoted 15 lines · Barry Song
Quoted 7 lines · David Carlier
sio_read_complete()usessio->pagesto account global PSWPIN vm events, butsio->pagestracks the number of bvec entries (folios), not base pages. For large folios this undercounts compared to the per-memcg path which correctly usesfolio_nr_pages(), and compared to the bdev read paths which also usefolio_nr_pages().Quoted 5 lines
Use
sio->len>>PAGE_SHIFTinstead, which gives the correct base page count sincesio->lenis accumulated viafolio_size(folio).Fixes a1a0dfd56f97 ("mm: handle THP in swap_*page_fs()")Signed-off-by David Carlier <[email protected]>
Quoted 6 lines
The patch seems theoretically correct, but I’m wondering where we can swap in mTHP for filesystem-based swap?
In both
do_swap_page()andshmem_swapin_folio(), we checkdata_race(si->flags & SWP_SYNCHRONOUS_IO)before allocating large folios. Am I missing something?
▎ The patch seems theoretically correct, but I'm wondering
▎ where we can swap in mTHP for filesystem-based swap?
▎ In both do_swap_page() and shmem_swapin_folio(), we check ▎ data_race(si->flags & SWP_SYNCHRONOUS_IO) before allocating ▎ large folios. Am I missing something?
You're right, I missed that. SWP_FS_OPS is only set by NFS and SMB which have no bdev, so SWP_SYNCHRONOUS_IO can never be set alongside it. Large folios can't currently reach this path since both do_swap_page() and shmem_swapin_folio() gate mTHP allocation on SWP_SYNCHRONOUS_IO.
That said, sio_read_complete() already calls count_mthp_stat() and the per-memcg accounting uses folio_nr_pages(), so the code seems written with large folios in mind even if the path is currently unreachable. Using sio->pages (bvec entry count) for a base-page count is still semantically wrong, but I understand the practical impact is nil today.
Happy to either drop this or keep it as a correctness cleanup, whatever you and Andrew prefer.
Cheers !
Quoted 24 lines
Quoted 19 lines
--- mm/page_io.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/page_io.c +1
-2@@ -497,7 +497,7 @@ static void sio_read_complete(struct kiocb *iocb, long ret)folio_mark_uptodate (folio);folio_unlock (folio); } -count_vm_events (PSWPIN , sio->pages); +count_vm_events (PSWPIN , sio->len >>PAGE_SHIFT ); }else {for (p =0 ; p < sio->pages; p++) {struct folio *folio =page_folio (sio->bvec[p].bv_page); --Quoted 2 lines
2.53.0
Thanks
Barry
Quoted 43 lines · David CARLIER
Quoted 15 lines · Barry Song
Quoted 7 lines · David Carlier
sio_read_complete()usessio->pagesto account global PSWPIN vm events, butsio->pagestracks the number of bvec entries (folios), not base pages. For large folios this undercounts compared to the per-memcg path which correctly usesfolio_nr_pages(), and compared to the bdev read paths which also usefolio_nr_pages().Quoted 5 lines
Use
sio->len>>PAGE_SHIFTinstead, which gives the correct base page count sincesio->lenis accumulated viafolio_size(folio).Fixes a1a0dfd56f97 ("mm: handle THP in swap_*page_fs()")Signed-off-by David Carlier <[email protected]>Quoted 6 lines
The patch seems theoretically correct, but I’m wondering where we can swap in mTHP for filesystem-based swap?
In both
do_swap_page()andshmem_swapin_folio(), we checkdata_race(si->flags & SWP_SYNCHRONOUS_IO)before allocating large folios. Am I missing something?▎ The patch seems theoretically correct, but I'm wondering
▎ where we can swap in mTHP for filesystem-based swap?▎ In both
do_swap_page()andshmem_swapin_folio(), we check ▎data_race(si->flags & SWP_SYNCHRONOUS_IO)before allocating ▎ large folios. Am I missing something?You're right, I missed that.
SWP_FS_OPSis only set by NFS and SMB which have no bdev, soSWP_SYNCHRONOUS_IOcan never be set alongside it. Large folios can't currently reach this path since bothdo_swap_page()andshmem_swapin_folio()gate mTHP allocation onSWP_SYNCHRONOUS_IO.That said,
sio_read_complete()already callscount_mthp_stat()and the per-memcg accounting usesfolio_nr_pages(), so the code seems written with large folios in mind even if the path is currently unreachable. Usingsio->pages(bvec entry count) for a base-page count is still semantically wrong, but I understand the practical impact is nil today.
Quoted 2 lines
Happy to either drop this or keep it as a correctness cleanup, whatever you and Andrew prefer.
Best to rework the patch description (clarify that it's a cleanup) and drop the Fixes: I guess.
Signature · 3 lines
Cheers, David
Barry Song
reviewerQuoted 44 lines · David CARLIER
Quoted 15 lines · Barry Song
Quoted 7 lines · David Carlier
sio_read_complete()usessio->pagesto account global PSWPIN vm events, butsio->pagestracks the number of bvec entries (folios), not base pages. For large folios this undercounts compared to the per-memcg path which correctly usesfolio_nr_pages(), and compared to the bdev read paths which also usefolio_nr_pages().Quoted 5 lines
Use
sio->len>>PAGE_SHIFTinstead, which gives the correct base page count sincesio->lenis accumulated viafolio_size(folio).Fixes a1a0dfd56f97 ("mm: handle THP in swap_*page_fs()")Signed-off-by David Carlier <[email protected]>Quoted 6 lines
The patch seems theoretically correct, but I’m wondering where we can swap in mTHP for filesystem-based swap?
In both
do_swap_page()andshmem_swapin_folio(), we checkdata_race(si->flags & SWP_SYNCHRONOUS_IO)before allocating large folios. Am I missing something?▎ The patch seems theoretically correct, but I'm wondering
▎ where we can swap in mTHP for filesystem-based swap?▎ In both
do_swap_page()andshmem_swapin_folio(), we check ▎data_race(si->flags & SWP_SYNCHRONOUS_IO)before allocating ▎ large folios. Am I missing something?You're right, I missed that.
SWP_FS_OPSis only set by NFS and SMB which have no bdev, soSWP_SYNCHRONOUS_IOcan never be set alongside it. Large folios can't currently reach this path since bothdo_swap_page()andshmem_swapin_folio()gate mTHP allocation onSWP_SYNCHRONOUS_IO.That said,
sio_read_complete()already callscount_mthp_stat()and the per-memcg accounting usesfolio_nr_pages(), so the code seems written with large folios in mind even if the path is currently unreachable. Usingsio->pages(bvec entry count) for a base-page count is still semantically wrong, but I understand the practical impact is nil today.
Quoted 3 lines
Happy to either drop this or keep it as a correctness cleanup, whatever you and Andrew prefer.
The patch looks correct, so I’d personally support keeping it after cleaning up the description, as David suggested.
Best Regards
Barry
PATCH v3 1/1 mm/page_io: use sio->len for PSWPIN accounting in sio_read_complete()
In reply to David Carliersio_read_complete() uses sio->pages to account global PSWPIN vm events, but sio->pages tracks the number of bvec entries (folios), not base pages.
While large folios cannot currently reach this path (SWP_FS_OPS and SWP_SYNCHRONOUS_IO are mutually exclusive, and mTHP swap-in allocation is gated on SWP_SYNCHRONOUS_IO), the accounting is semantically inconsistent with the per-memcg path which correctly uses folio_nr_pages().
Use sio->len >> PAGE_SHIFT instead, which gives the correct base page count since sio->len is accumulated via folio_size(folio).
--- mm/page_io.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/page_io.c +1 -1
@@ -497,7 +497,7 @@ static void sio_read_complete(struct kiocb *iocb, long ret)folio_mark_uptodate (folio);folio_unlock (folio); } -count_vm_events (PSWPIN , sio->pages); +count_vm_events (PSWPIN , sio->len >>PAGE_SHIFT ); }else {for (p =0 ; p < sio->pages; p++) {struct folio *folio =page_folio (sio->bvec[p].bv_page);
Signature · 1 line
2.53.0
Re: [PATCH v3] mm/page_io: use sio->len for PSWPIN accounting in sio_read_complete()
Quoted 10 lines · David Carlier
sio_read_complete()usessio->pagesto account global PSWPIN vm events, butsio->pagestracks the number of bvec entries (folios), not base pages.While large folios cannot currently reach this path (
SWP_FS_OPSandSWP_SYNCHRONOUS_IOare mutually exclusive, and mTHP swap-in allocation is gated onSWP_SYNCHRONOUS_IO), the accounting is semantically inconsistent with the per-memcg path which correctly usesfolio_nr_pages().
Quoted 4 lines
Use
sio->len>>PAGE_SHIFTinstead, which gives the correct base page count sincesio->lenis accumulated viafolio_size(folio).Signed-off-by David Carlier <[email protected]>
For the next time: Please don't send new revisions as reply to other revisions :)
Quoted 9 lines
--- mm/page_io.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/page_io.c +0
-0
mm/page_io.c static void sio_read_complete(struct kiocb *iocb, long ret) · line 503
Quoted 8 lines
mm/page_io.c +1
-1@@ -497,7 +497,7 @@ static void sio_read_complete(struct kiocb *iocb, long ret)folio_mark_uptodate (folio);folio_unlock (folio); } -count_vm_events (PSWPIN , sio->pages); +count_vm_events (PSWPIN , sio->len >>PAGE_SHIFT ); }else {for (p =0 ; p < sio->pages; p++) {struct folio *folio =page_folio (sio->bvec[p].bv_page);
sio->len always covers full pages as processed in swap_read_folio_fs(), so there should not be any difference.
Signature · 3 lines
Cheers, David
Andrew Morton
maintainerRe: [PATCH v3] mm/page_io: use sio->len for PSWPIN accounting in sio_read_complete()
Quoted 11 lines · On Wed, 1 Apr 2026 10:32:30 +0200 "David Hildenbrand (Arm)" <[email protected]>…
Quoted 10 lines · David Carlier
sio_read_complete()usessio->pagesto account global PSWPIN vm events, butsio->pagestracks the number of bvec entries (folios), not base pages.While large folios cannot currently reach this path (
SWP_FS_OPSandSWP_SYNCHRONOUS_IOare mutually exclusive, and mTHP swap-in allocation is gated onSWP_SYNCHRONOUS_IO), the accounting is semantically inconsistent with the per-memcg path which correctly usesfolio_nr_pages().
Quoted 7 lines
Quoted 4 lines
Use
sio->len>>PAGE_SHIFTinstead, which gives the correct base page count sincesio->lenis accumulated viafolio_size(folio).Signed-off-by David Carlier <[email protected]>For the next time: Please don't send new revisions as reply to other revisions :)
Also, altering the Subject: causes akpm confusion (not hard). But having accurate titles is more important than avoiding akpm confusion, so there is that.
A little "what i changed since v2" note after the "---" separator is helpful, please.
But I figured it out! Thanks, I'll pop this into my after-rc1 pile.
Re: [PATCH v3] mm/page_io: use sio->len for PSWPIN accounting in sio_read_complete()
In reply to David CarlierQuoted 11 lines · David Carlier
sio_read_complete()usessio->pagesto account global PSWPIN vm events, butsio->pagestracks the number of bvec entries (folios), not base pages.While large folios cannot currently reach this path (
SWP_FS_OPSandSWP_SYNCHRONOUS_IOare mutually exclusive, and mTHP swap-in allocation is gated onSWP_SYNCHRONOUS_IO), the accounting is semantically inconsistent with the per-memcg path which correctly usesfolio_nr_pages().
Quoted 4 lines
Use
sio->len>>PAGE_SHIFTinstead, which gives the correct base page count sincesio->lenis accumulated viafolio_size(folio).Signed-off-by David Carlier <[email protected]>
LGTM,
Re: [PATCH v3] mm/page_io: use sio->len for PSWPIN accounting in sio_read_complete()
In reply to David CarlierQuoted 11 lines · David Carlier
sio_read_complete()usessio->pagesto account global PSWPIN vm events, butsio->pagestracks the number of bvec entries (folios), not base pages.While large folios cannot currently reach this path (
SWP_FS_OPSandSWP_SYNCHRONOUS_IOare mutually exclusive, and mTHP swap-in allocation is gated onSWP_SYNCHRONOUS_IO), the accounting is semantically inconsistent with the per-memcg path which correctly usesfolio_nr_pages().
Quoted 4 lines
Use
sio->len>>PAGE_SHIFTinstead, which gives the correct base page count sincesio->lenis accumulated viafolio_size(folio).Signed-off-by David Carlier <[email protected]>
Chris
Re: [PATCH v3] mm/page_io: use sio->len for PSWPIN accounting in sio_read_complete()
In reply to David CarlierQuoted 3 lines · David Carlier
sio_read_complete()usessio->pagesto account global PSWPIN vm events, butsio->pagestracks the number of bvec entries (folios), not base pages.
Since you're messing with this file anyway, do you want to submit a third patch to rename swap_iocb->pages to ->bvecs?
Barry Song
reviewerRe: [PATCH v3] mm/page_io: use sio->len for PSWPIN accounting in sio_read_complete()
Quoted 8 lines · Matthew Wilcox
Quoted 3 lines · David Carlier
sio_read_complete()usessio->pagesto account global PSWPIN vm events, butsio->pagestracks the number of bvec entries (folios), not base pages.Since you're messing with this file anyway, do you want to submit a third patch to rename
swap_iocb->pagesto ->bvecs?
+1, that would be great.
Re: [PATCH v3] mm/page_io: use sio->len for PSWPIN accounting in sio_read_complete()
In reply to Matthew WilcoxSure, will send as a follow-up patch.
Cheers.
Quoted 8 lines · Matthew Wilcox
Quoted 3 lines · David Carlier
sio_read_complete()usessio->pagesto account global PSWPIN vm events, butsio->pagestracks the number of bvec entries (folios), not base pages.Since you're messing with this file anyway, do you want to submit a third patch to rename
swap_iocb->pagesto ->bvecs?