mm/page_io: fix PSWPIN undercount for large folios in sio_read_complete()
Subsystems (2)
PATCH v2 1/1 mm/page_io: fix PSWPIN undercount for large folios in sio_read_complete()
sio_read_complete() uses sio->pages to account global PSWPIN vm events, but sio->pages tracks the number of bvec entries (folios), not base pages. For large folios this undercounts compared to the per-memcg path which correctly uses folio_nr_pages(), and compared to the bdev read paths which also use folio_nr_pages().
Use sio->len >> PAGE_SHIFT instead, which gives the correct base page count since sio->len is accumulated via folio_size(folio).
--- mm/page_io.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/page_io.c +1 -1
@@ -497,7 +497,7 @@ static void sio_read_complete(struct kiocb *iocb, long ret)folio_mark_uptodate (folio);folio_unlock (folio); } -count_vm_events (PSWPIN , sio->pages); +count_vm_events (PSWPIN , sio->len >>PAGE_SHIFT ); }else {for (p =0 ; p < sio->pages; p++) {struct folio *folio =page_folio (sio->bvec[p].bv_page);
Signature · 1 line
2.53.0
Andrew Morton
maintainerQuoted 8 lines · On Mon, 30 Mar 2026 08:12:29 +0100 David Carlier <[email protected]> wrote:
sio_read_complete()usessio->pagesto account global PSWPIN vm events, butsio->pagestracks the number of bvec entries (folios), not base pages. For large folios this undercounts compared to the per-memcg path which correctly usesfolio_nr_pages(), and compared to the bdev read paths which also usefolio_nr_pages().Use
sio->len>>PAGE_SHIFTinstead, which gives the correct base page count sincesio->lenis accumulated viafolio_size(folio).
Thanks. What are the userspace-visible runtime effects of the bug? Just /proc/vmstat wrongness?
Seems pretty minor, and it's been in there for several years. But I guess I'll slip a cc:stable in there, let it trickle back at a later time.
Quoted 14 lines
mm/page_io.c +1
-1@@ -497,7 +497,7 @@ static void sio_read_complete(struct kiocb *iocb, long ret)folio_mark_uptodate (folio);folio_unlock (folio); } -count_vm_events (PSWPIN , sio->pages); +count_vm_events (PSWPIN , sio->len >>PAGE_SHIFT ); }else {for (p =0 ; p < sio->pages; p++) {struct folio *folio =page_folio (sio->bvec[p].bv_page);Signature · 1 line
2.53.0
Quoted 7 lines · David Carlier
sio_read_complete()usessio->pagesto account global PSWPIN vm events, butsio->pagestracks the number of bvec entries (folios), not base pages. For large folios this undercounts compared to the per-memcg path which correctly usesfolio_nr_pages(), and compared to the bdev read paths which also usefolio_nr_pages().
Quoted 5 lines
Use
sio->len>>PAGE_SHIFTinstead, which gives the correct base page count sincesio->lenis accumulated viafolio_size(folio).Fixes a1a0dfd56f97 ("mm: handle THP in swap_*page_fs()")Signed-off-by David Carlier <[email protected]>
The patch seems theoretically correct, but I’m wondering where we can swap in mTHP for filesystem-based swap?
In both do_swap_page() and shmem_swapin_folio(), we check data_race(si->flags & SWP_SYNCHRONOUS_IO) before allocating large folios. Am I missing something?
Quoted 18 lines
--- mm/page_io.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/page_io.c +1
-2@@ -497,7 +497,7 @@ static void sio_read_complete(struct kiocb *iocb, long ret)folio_mark_uptodate (folio);folio_unlock (folio); } -count_vm_events (PSWPIN , sio->pages); +count_vm_events (PSWPIN , sio->len >>PAGE_SHIFT ); }else {for (p =0 ; p < sio->pages; p++) {struct folio *folio =page_folio (sio->bvec[p].bv_page); --
Quoted 2 lines
2.53.0
Thanks
Barry
Quoted 15 lines · Barry Song
Quoted 7 lines · David Carlier
sio_read_complete()usessio->pagesto account global PSWPIN vm events, butsio->pagestracks the number of bvec entries (folios), not base pages. For large folios this undercounts compared to the per-memcg path which correctly usesfolio_nr_pages(), and compared to the bdev read paths which also usefolio_nr_pages().Quoted 5 lines
Use
sio->len>>PAGE_SHIFTinstead, which gives the correct base page count sincesio->lenis accumulated viafolio_size(folio).Fixes a1a0dfd56f97 ("mm: handle THP in swap_*page_fs()")Signed-off-by David Carlier <[email protected]>
Quoted 6 lines
The patch seems theoretically correct, but I’m wondering where we can swap in mTHP for filesystem-based swap?
In both
do_swap_page()andshmem_swapin_folio(), we checkdata_race(si->flags & SWP_SYNCHRONOUS_IO)before allocating large folios. Am I missing something?
▎ The patch seems theoretically correct, but I'm wondering
▎ where we can swap in mTHP for filesystem-based swap?
▎ In both do_swap_page() and shmem_swapin_folio(), we check ▎ data_race(si->flags & SWP_SYNCHRONOUS_IO) before allocating ▎ large folios. Am I missing something?
You're right, I missed that. SWP_FS_OPS is only set by NFS and SMB which have no bdev, so SWP_SYNCHRONOUS_IO can never be set alongside it. Large folios can't currently reach this path since both do_swap_page() and shmem_swapin_folio() gate mTHP allocation on SWP_SYNCHRONOUS_IO.
That said, sio_read_complete() already calls count_mthp_stat() and the per-memcg accounting uses folio_nr_pages(), so the code seems written with large folios in mind even if the path is currently unreachable. Using sio->pages (bvec entry count) for a base-page count is still semantically wrong, but I understand the practical impact is nil today.
Happy to either drop this or keep it as a correctness cleanup, whatever you and Andrew prefer.
Cheers !
Quoted 24 lines
Quoted 19 lines
--- mm/page_io.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/page_io.c +1
-2@@ -497,7 +497,7 @@ static void sio_read_complete(struct kiocb *iocb, long ret)folio_mark_uptodate (folio);folio_unlock (folio); } -count_vm_events (PSWPIN , sio->pages); +count_vm_events (PSWPIN , sio->len >>PAGE_SHIFT ); }else {for (p =0 ; p < sio->pages; p++) {struct folio *folio =page_folio (sio->bvec[p].bv_page); --Quoted 2 lines
2.53.0
Thanks
Barry
Quoted 43 lines · David CARLIER
Quoted 15 lines · Barry Song
Quoted 7 lines · David Carlier
sio_read_complete()usessio->pagesto account global PSWPIN vm events, butsio->pagestracks the number of bvec entries (folios), not base pages. For large folios this undercounts compared to the per-memcg path which correctly usesfolio_nr_pages(), and compared to the bdev read paths which also usefolio_nr_pages().Quoted 5 lines
Use
sio->len>>PAGE_SHIFTinstead, which gives the correct base page count sincesio->lenis accumulated viafolio_size(folio).Fixes a1a0dfd56f97 ("mm: handle THP in swap_*page_fs()")Signed-off-by David Carlier <[email protected]>Quoted 6 lines
The patch seems theoretically correct, but I’m wondering where we can swap in mTHP for filesystem-based swap?
In both
do_swap_page()andshmem_swapin_folio(), we checkdata_race(si->flags & SWP_SYNCHRONOUS_IO)before allocating large folios. Am I missing something?▎ The patch seems theoretically correct, but I'm wondering
▎ where we can swap in mTHP for filesystem-based swap?▎ In both
do_swap_page()andshmem_swapin_folio(), we check ▎data_race(si->flags & SWP_SYNCHRONOUS_IO)before allocating ▎ large folios. Am I missing something?You're right, I missed that.
SWP_FS_OPSis only set by NFS and SMB which have no bdev, soSWP_SYNCHRONOUS_IOcan never be set alongside it. Large folios can't currently reach this path since bothdo_swap_page()andshmem_swapin_folio()gate mTHP allocation onSWP_SYNCHRONOUS_IO.That said,
sio_read_complete()already callscount_mthp_stat()and the per-memcg accounting usesfolio_nr_pages(), so the code seems written with large folios in mind even if the path is currently unreachable. Usingsio->pages(bvec entry count) for a base-page count is still semantically wrong, but I understand the practical impact is nil today.
Quoted 2 lines
Happy to either drop this or keep it as a correctness cleanup, whatever you and Andrew prefer.
Best to rework the patch description (clarify that it's a cleanup) and drop the Fixes: I guess.
Signature · 3 lines
Cheers, David
PATCH v3 1/1 mm/page_io: use sio->len for PSWPIN accounting in sio_read_complete()
In reply to David Carliersio_read_complete() uses sio->pages to account global PSWPIN vm events, but sio->pages tracks the number of bvec entries (folios), not base pages.
While large folios cannot currently reach this path (SWP_FS_OPS and SWP_SYNCHRONOUS_IO are mutually exclusive, and mTHP swap-in allocation is gated on SWP_SYNCHRONOUS_IO), the accounting is semantically inconsistent with the per-memcg path which correctly uses folio_nr_pages().
Use sio->len >> PAGE_SHIFT instead, which gives the correct base page count since sio->len is accumulated via folio_size(folio).
--- mm/page_io.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/page_io.c +1 -1
@@ -497,7 +497,7 @@ static void sio_read_complete(struct kiocb *iocb, long ret)folio_mark_uptodate (folio);folio_unlock (folio); } -count_vm_events (PSWPIN , sio->pages); +count_vm_events (PSWPIN , sio->len >>PAGE_SHIFT ); }else {for (p =0 ; p < sio->pages; p++) {struct folio *folio =page_folio (sio->bvec[p].bv_page);
Signature · 1 line
2.53.0
Re: [PATCH v3] mm/page_io: use sio->len for PSWPIN accounting in sio_read_complete()
Quoted 10 lines · David Carlier
sio_read_complete()usessio->pagesto account global PSWPIN vm events, butsio->pagestracks the number of bvec entries (folios), not base pages.While large folios cannot currently reach this path (
SWP_FS_OPSandSWP_SYNCHRONOUS_IOare mutually exclusive, and mTHP swap-in allocation is gated onSWP_SYNCHRONOUS_IO), the accounting is semantically inconsistent with the per-memcg path which correctly usesfolio_nr_pages().
Quoted 4 lines
Use
sio->len>>PAGE_SHIFTinstead, which gives the correct base page count sincesio->lenis accumulated viafolio_size(folio).Signed-off-by David Carlier <[email protected]>
For the next time: Please don't send new revisions as reply to other revisions :)
Quoted 9 lines
--- mm/page_io.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/page_io.c +0
-0
mm/page_io.c static void sio_read_complete(struct kiocb *iocb, long ret) · line 503
Quoted 8 lines
mm/page_io.c +1
-1@@ -497,7 +497,7 @@ static void sio_read_complete(struct kiocb *iocb, long ret)folio_mark_uptodate (folio);folio_unlock (folio); } -count_vm_events (PSWPIN , sio->pages); +count_vm_events (PSWPIN , sio->len >>PAGE_SHIFT ); }else {for (p =0 ; p < sio->pages; p++) {struct folio *folio =page_folio (sio->bvec[p].bv_page);
sio->len always covers full pages as processed in swap_read_folio_fs(), so there should not be any difference.
Signature · 3 lines
Cheers, David
Barry Song
reviewerQuoted 44 lines · David CARLIER
Quoted 15 lines · Barry Song
Quoted 7 lines · David Carlier
sio_read_complete()usessio->pagesto account global PSWPIN vm events, butsio->pagestracks the number of bvec entries (folios), not base pages. For large folios this undercounts compared to the per-memcg path which correctly usesfolio_nr_pages(), and compared to the bdev read paths which also usefolio_nr_pages().Quoted 5 lines
Use
sio->len>>PAGE_SHIFTinstead, which gives the correct base page count sincesio->lenis accumulated viafolio_size(folio).Fixes a1a0dfd56f97 ("mm: handle THP in swap_*page_fs()")Signed-off-by David Carlier <[email protected]>Quoted 6 lines
The patch seems theoretically correct, but I’m wondering where we can swap in mTHP for filesystem-based swap?
In both
do_swap_page()andshmem_swapin_folio(), we checkdata_race(si->flags & SWP_SYNCHRONOUS_IO)before allocating large folios. Am I missing something?▎ The patch seems theoretically correct, but I'm wondering
▎ where we can swap in mTHP for filesystem-based swap?▎ In both
do_swap_page()andshmem_swapin_folio(), we check ▎data_race(si->flags & SWP_SYNCHRONOUS_IO)before allocating ▎ large folios. Am I missing something?You're right, I missed that.
SWP_FS_OPSis only set by NFS and SMB which have no bdev, soSWP_SYNCHRONOUS_IOcan never be set alongside it. Large folios can't currently reach this path since bothdo_swap_page()andshmem_swapin_folio()gate mTHP allocation onSWP_SYNCHRONOUS_IO.That said,
sio_read_complete()already callscount_mthp_stat()and the per-memcg accounting usesfolio_nr_pages(), so the code seems written with large folios in mind even if the path is currently unreachable. Usingsio->pages(bvec entry count) for a base-page count is still semantically wrong, but I understand the practical impact is nil today.
Quoted 3 lines
Happy to either drop this or keep it as a correctness cleanup, whatever you and Andrew prefer.
The patch looks correct, so I’d personally support keeping it after cleaning up the description, as David suggested.
Best Regards
Barry
Andrew Morton
maintainerRe: [PATCH v3] mm/page_io: use sio->len for PSWPIN accounting in sio_read_complete()
In reply to David Hildenbrand (Arm)Quoted 11 lines · On Wed, 1 Apr 2026 10:32:30 +0200 "David Hildenbrand (Arm)" <[email protected]>…
Quoted 10 lines · David Carlier
sio_read_complete()usessio->pagesto account global PSWPIN vm events, butsio->pagestracks the number of bvec entries (folios), not base pages.While large folios cannot currently reach this path (
SWP_FS_OPSandSWP_SYNCHRONOUS_IOare mutually exclusive, and mTHP swap-in allocation is gated onSWP_SYNCHRONOUS_IO), the accounting is semantically inconsistent with the per-memcg path which correctly usesfolio_nr_pages().
Quoted 7 lines
Quoted 4 lines
Use
sio->len>>PAGE_SHIFTinstead, which gives the correct base page count sincesio->lenis accumulated viafolio_size(folio).Signed-off-by David Carlier <[email protected]>For the next time: Please don't send new revisions as reply to other revisions :)
Also, altering the Subject: causes akpm confusion (not hard). But having accurate titles is more important than avoiding akpm confusion, so there is that.
A little "what i changed since v2" note after the "---" separator is helpful, please.
But I figured it out! Thanks, I'll pop this into my after-rc1 pile.
Re: [PATCH v3] mm/page_io: use sio->len for PSWPIN accounting in sio_read_complete()
In reply to David CarlierQuoted 11 lines · David Carlier
sio_read_complete()usessio->pagesto account global PSWPIN vm events, butsio->pagestracks the number of bvec entries (folios), not base pages.While large folios cannot currently reach this path (
SWP_FS_OPSandSWP_SYNCHRONOUS_IOare mutually exclusive, and mTHP swap-in allocation is gated onSWP_SYNCHRONOUS_IO), the accounting is semantically inconsistent with the per-memcg path which correctly usesfolio_nr_pages().
Quoted 4 lines
Use
sio->len>>PAGE_SHIFTinstead, which gives the correct base page count sincesio->lenis accumulated viafolio_size(folio).Signed-off-by David Carlier <[email protected]>
LGTM,
Re: [PATCH v3] mm/page_io: use sio->len for PSWPIN accounting in sio_read_complete()
In reply to David CarlierQuoted 11 lines · David Carlier
sio_read_complete()usessio->pagesto account global PSWPIN vm events, butsio->pagestracks the number of bvec entries (folios), not base pages.While large folios cannot currently reach this path (
SWP_FS_OPSandSWP_SYNCHRONOUS_IOare mutually exclusive, and mTHP swap-in allocation is gated onSWP_SYNCHRONOUS_IO), the accounting is semantically inconsistent with the per-memcg path which correctly usesfolio_nr_pages().
Quoted 4 lines
Use
sio->len>>PAGE_SHIFTinstead, which gives the correct base page count sincesio->lenis accumulated viafolio_size(folio).Signed-off-by David Carlier <[email protected]>
Chris
Re: [PATCH v3] mm/page_io: use sio->len for PSWPIN accounting in sio_read_complete()
In reply to David CarlierQuoted 3 lines · David Carlier
sio_read_complete()usessio->pagesto account global PSWPIN vm events, butsio->pagestracks the number of bvec entries (folios), not base pages.
Since you're messing with this file anyway, do you want to submit a third patch to rename swap_iocb->pages to ->bvecs?
Barry Song
reviewerRe: [PATCH v3] mm/page_io: use sio->len for PSWPIN accounting in sio_read_complete()
Quoted 8 lines · Matthew Wilcox
Quoted 3 lines · David Carlier
sio_read_complete()usessio->pagesto account global PSWPIN vm events, butsio->pagestracks the number of bvec entries (folios), not base pages.Since you're messing with this file anyway, do you want to submit a third patch to rename
swap_iocb->pagesto ->bvecs?
+1, that would be great.
Re: [PATCH v3] mm/page_io: use sio->len for PSWPIN accounting in sio_read_complete()
In reply to Matthew WilcoxSure, will send as a follow-up patch.
Cheers.
Quoted 8 lines · Matthew Wilcox
Quoted 3 lines · David Carlier
sio_read_complete()usessio->pagesto account global PSWPIN vm events, butsio->pagestracks the number of bvec entries (folios), not base pages.Since you're messing with this file anyway, do you want to submit a third patch to rename
swap_iocb->pagesto ->bvecs?