better block swap batching and a different take on swap_ops v5
Subsystems (16)
Hi all,
this series makes use of the swap_iocb for block as well so that it doesn't do inefficient single-bio I/O, and then rebases the swap_ops from Baoquan on top of the now very different method structure.
When running doing kernels builds, which is a workload that doesn't really do much THP anonymous memory it still gets 2x clustering for writeout and 1.2x for reading back swap in. The overall times do not actually change, though.
Changes since v4:
- really fix the charged vs non-charged merging decision
Changes since v3:
- rebase to latest mm-unstable
- don't merge charged and non-charged folios into a single bio or fs swap request
Changes since v2:
- rebased to 7.2-rc2
- rename swap_add_page to swap_add_folio
- rename SWAP_OPS_F_NOFS to SWAP_OPS_F_REQUIRE_NOFS and document it
- improve various commit logs
- spelling fixes
Changes since v1:
- drop the memcg accounting for fs backed swap as it can't really work as-is
- change the related refactoring to not create a bisection hazard or compile issue with the wrong set of cgroup config options enabled
- remove a now dead function
Diffstat:
Documentation/filesystems/locking.rst | 5 Documentation/filesystems/vfs.rst | 4 drivers/gpu/drm/i915/gem/i915_gem_shmem.c | 2 drivers/gpu/drm/ttm/ttm_backup.c | 2 fs/nfs/file.c | 4 fs/smb/client/file.c | 4 include/linux/shmem_fs.h | 5 include/linux/swap.h | 7 include/linux/vm_event_item.h | 4 mm/madvise.c | 16 mm/page_io.c | 586 +++++++++++++++--------------- mm/shmem.c | 18 mm/swap.h | 67 +-- mm/swap_state.c | 53 +- mm/swapfile.c | 11 mm/vmscan.c | 30 - mm/vmstat.c | 6 mm/zswap.c | 4 18 files changed, 432 insertions(+), 396 deletions(-)
PATCH 1/7 shmem: provide a shmem_write_folio wrapper
Provide a wrapper for the shmem abuses in drm to prepare for swap I/O refactoring by keeping swap_iocb handling entirely contained in mm/.
--- drivers/gpu/drm/i915/gem/i915_gem_shmem.c | 2 +- drivers/gpu/drm/ttm/ttm_backup.c | 2 +- include/linux/shmem_fs.h | 5 +---- mm/shmem.c | 7 ++++++- mm/swap.h | 4 ++++ 5 files changed, 13 insertions(+), 7 deletions(-)
drivers/gpu/drm/i915/gem/i915_gem_shmem.c +1 -1
@@ -325,7 +325,7 @@ void __shmem_writeback(size_t size, struct address_space *mapping)if (folio_mapped (folio))folio_redirty_for_writepage (&wbc, folio);else - error =shmem_writeout (folio,NULL ,NULL ); + error =shmem_write_folio (folio); } }
drivers/gpu/drm/ttm/ttm_backup.c +1 -1
@@ -117,7 +117,7 @@ ttm_backup_backup_page(struct file *backup, struct page *page,if (writeback && !folio_mapped (to_folio) &&folio_clear_dirty_for_io (to_folio)) {folio_set_reclaim (to_folio); - ret =shmem_writeout (to_folio,NULL ,NULL ); + ret =shmem_write_folio (to_folio);if (!folio_test_writeback (to_folio))folio_clear_reclaim (to_folio);/*
include/linux/shmem_fs.h +1 -4
@@ -12,8 +12,6 @@#include <linux/userfaultfd_k.h> #include <linux/bits.h> -struct swap_iocb ; -/* inode in-kernel data */ #ifdef CONFIG_TMPFS_QUOTA
@@ -123,8 +121,7 @@ static inline bool shmem_mapping(const struct address_space *mapping)void shmem_unlock_mapping (struct address_space *mapping);struct page *shmem_read_mapping_page_gfp (struct address_space *mapping,pgoff_t index,gfp_t gfp_mask); -int shmem_writeout (struct folio *folio,struct swap_iocb **plug, -struct list_head *folio_list); +int shmem_write_folio (struct folio *folio);void shmem_truncate_range (struct inode *inode,loff_t start,uoff_t end);int shmem_unuse (unsigned int type);
mm/shmem.c +6 -1
@@ -1737,7 +1737,12 @@ int shmem_writeout(struct folio *folio, struct swap_iocb **plug,folio_mark_dirty (folio);return AOP_WRITEPAGE_ACTIVATE ;/* Return with folio locked */ } -EXPORT_SYMBOL_GPL (shmem_writeout); + +int shmem_write_folio (struct folio *folio) +{ +return shmem_writeout (folio,NULL ,NULL ); +} +EXPORT_SYMBOL_GPL (shmem_write_folio);#if defined(CONFIG_NUMA ) && defined(CONFIG_TMPFS )static void shmem_show_mpol (struct seq_file *seq,struct mempolicy *mpol)
mm/swap.h +4 -0
@@ -471,4 +471,8 @@ static inline unsigned int folio_swap_flags(struct folio *folio) }#endif /* CONFIG_SWAP */ + +int shmem_writeout (struct folio *folio,struct swap_iocb **plug, +struct list_head *folio_list); +#endif /* _MM_SWAP_H */
Signature · 1 line
2.53.0
Re: [PATCH 1/7] shmem: provide a shmem_write_folio wrapper
Quoted 8 lines · Christoph Hellwig
Provide a wrapper for the shmem abuses in drm to prepare for swap I/O refactoring by keeping
swap_iocbhandling entirely contained inmm/.Signed-off-by Christoph Hellwig <[email protected]>Reviewed-by Baoquan He <[email protected]>Reviewed-by Nhat Pham <[email protected]>Reviewed-by Baolin Wang <[email protected]>Acked-by Chris Li <[email protected]>
Quoted 7 lines
--- drivers/gpu/drm/i915/gem/i915_gem_shmem.c | 2 +- drivers/gpu/drm/ttm/ttm_backup.c | 2 +- include/linux/shmem_fs.h | 5 +---- mm/shmem.c | 7 ++++++- mm/swap.h | 4 ++++ 5 files changed, 13 insertions(+), 7 deletions(-)
PATCH 2/7 mm/swap: introduce struct swap_io_ctx
In reply to Christoph HellwigGeneralize the context currently provided by double pointers to struct swap_iocb to an on-stack context. This cleans up the code and prepares for adding more fields and supporting batching multiple folios into a single bio for block-based swap as well.
This new swap_io_ctx is required for all functions using it, the old way of allowing a NULL iocb for some callers is removed to keep the interface consistent. To reduce code duplication caused by this, a new swap_cache_read_folio_sync helper is added to consolidate the code to call swap_cache_read_folio with a local swap_io_ctx.
The unpug helpers are renamed to use the submit wording as they are generalized.
--- mm/madvise.c | 16 ++++++------- mm/page_io.c | 60 ++++++++++++++++++++++++++----------------------- mm/shmem.c | 13 +++++++---- mm/swap.h | 36 +++++++++++++---------------- mm/swap_state.c | 53 +++++++++++++++++++++++++------------------ mm/vmscan.c | 15 ++++++------- mm/zswap.c | 4 +++- 7 files changed, 106 insertions(+), 91 deletions(-)
mm/madvise.c +8 -8
@@ -188,7 +188,7 @@ static int swapin_walk_pmd_entry(pmd_t *pmd, unsigned long start, unsigned long end, struct mm_walk *walk) { struct vm_area_struct *vma = walk->private; -struct swap_iocb *splug =NULL ; +struct swap_io_ctx ctx = {};pte_t *ptep =NULL ;spinlock_t *ptl;unsigned long addr;
@@ -212,15 +212,15 @@ static int swapin_walk_pmd_entry(pmd_t *pmd, unsigned long start,pte_unmap_unlock (ptep, ptl); ptep =NULL ; - folio =read_swap_cache_async (entry,GFP_HIGHUSER_MOVABLE , - vma, addr, &splug); + folio =read_swap_cache_async (&ctx, entry,GFP_HIGHUSER_MOVABLE , + vma, addr);if (folio)folio_put (folio); }if (ptep)pte_unmap_unlock (ptep, ptl); -swap_read_unplug (splug); +swap_read_submit (&ctx);cond_resched ();return 0 ;
@@ -238,7 +238,7 @@ static void shmem_swapin_range(struct vm_area_struct *vma,XA_STATE (xas, &mapping->i_pages,linear_page_index (vma, start));pgoff_t end_index =linear_page_index (vma, end) -1 ;struct folio *folio; -struct swap_iocb *splug =NULL ; +struct swap_io_ctx ctx = {};rcu_read_lock ();xas_for_each (&xas, folio, end_index) {
@@ -257,15 +257,15 @@ static void shmem_swapin_range(struct vm_area_struct *vma,xas_pause (&xas);rcu_read_unlock (); - folio =read_swap_cache_async (entry,mapping_gfp_mask (mapping), - vma, addr, &splug); + folio =read_swap_cache_async (&ctx, entry, +mapping_gfp_mask (mapping), vma, addr);if (folio)folio_put (folio); rcu_read_lock(); }rcu_read_unlock (); -swap_read_unplug (splug); +swap_read_submit (&ctx); }#endif /* CONFIG_SWAP */
mm/page_io.c +32 -28
162 lines not on this page · raw message
mm/shmem.c +9 -4
@@ -1583,13 +1583,13 @@ int shmem_unuse(unsigned int type)/** * shmem_writeout - Write the folio to swap +* @ctx: swap I/O context * @folio: The folio to write -* @plug: swap plug * @folio_list: list to put back folios on split * * Move the folio from the page cache to the swap cache. */ -int shmem_writeout (struct folio *folio,struct swap_iocb **plug, +int shmem_writeout (struct swap_io_ctx *ctx,struct folio *folio,struct list_head *folio_list) {struct address_space *mapping = folio->mapping;
@@ -1701,7 +1701,7 @@ int shmem_writeout(struct folio *folio, struct swap_iocb **plug,shmem_delete_from_page_cache (folio,swp_to_radix_entry (folio->swap));BUG_ON (folio_mapped (folio)); - error =swap_writeout (folio, plug); + error =swap_writeout (ctx, folio);if (error !=AOP_WRITEPAGE_ACTIVATE ) {/* folio has been unlocked */ return error;
@@ -1740,7 +1740,12 @@ int shmem_writeout(struct folio *folio, struct swap_iocb **plug,int shmem_write_folio (struct folio *folio) { -return shmem_writeout (folio,NULL ,NULL ); +struct swap_io_ctx ctx = {}; +int err; + + err =shmem_writeout (&ctx, folio,NULL ); +swap_write_submit (&ctx); +return err; }EXPORT_SYMBOL_GPL (shmem_write_folio);
mm/swap.h +16 -20
@@ -4,6 +4,7 @@#include <linux/atomic.h> /* for atomic_long_t */ #include <linux/mm.h> /* for PAGE_SHIFT */ +struct mempolicy ;struct swap_iocb ;struct swap_memcg_table ;
@@ -76,6 +77,10 @@ enum swap_cluster_flags { CLUSTER_FLAG_MAX, }; +struct swap_io_ctx { +struct swap_iocb *sio; +}; +#ifdef CONFIG_SWAP #include <linux/swapops.h> /* for swp_offset */ #include <linux/blk_types.h> /* for bio_end_io_t */
@@ -238,17 +243,11 @@ extern void __swap_cluster_free_entries(struct swap_info_struct *si,/* linux/mm/page_io.c */ int sio_pool_init (void ); -struct swap_iocb ; -void swap_read_folio (struct folio *folio,struct swap_iocb **plug); -void __swap_read_unplug (struct swap_iocb *plug); -static inline void swap_read_unplug (struct swap_iocb *plug) -{ -if (unlikely (plug)) -__swap_read_unplug (plug); -} -void swap_write_unplug (struct swap_iocb *sio); -int swap_writeout (struct folio *folio,struct swap_iocb **swap_plug); -void __swap_writepage (struct folio *folio,struct swap_iocb **swap_plug); +void swap_read_folio (struct swap_io_ctx *ctx,struct folio *folio); +void swap_read_submit (struct swap_io_ctx *ctx); +void swap_write_submit (struct swap_io_ctx *ctx); +int swap_writeout (struct swap_io_ctx *ctx,struct folio *folio); +void __swap_writepage (struct swap_io_ctx *ctx,struct folio *folio);/* linux/mm/swap_state.c */ extern struct address_space swap_space__read_mostly ;
@@ -315,9 +314,8 @@ void __swap_cache_replace_folio(struct swap_cluster_info *ci,void show_swap_cache_info (void );void swapcache_clear (struct swap_info_struct *si,swp_entry_t entry,int nr); -struct folio *read_swap_cache_async (swp_entry_t entry,gfp_t gfp_mask, -struct vm_area_struct *vma,unsigned long addr, -struct swap_iocb **plug); +struct folio *read_swap_cache_async (struct swap_io_ctx *ctx,swp_entry_t entry, +gfp_t gfp_mask,struct vm_area_struct *vma,unsigned long addr);struct folio *swap_cluster_readahead (swp_entry_t entry,gfp_t flag,struct mempolicy *mpol,pgoff_t ilx); struct folio *swapin_readahead(swp_entry_t entry, gfp_t flag,
@@ -333,7 +331,6 @@ static inline unsigned int folio_swap_flags(struct folio *folio) }#else /* CONFIG_SWAP */ -struct swap_iocb ;static inline struct swap_cluster_info *swap_cluster_lock (struct swap_info_struct *si,pgoff_t offset,bool irq) {
@@ -379,11 +376,11 @@ static inline void folio_put_swap(struct folio *folio, struct page *page) { } -static inline void swap_read_folio (struct folio *folio,struct swap_iocb **plug) +static inline void swap_read_folio (struct swap_io_ctx *ctx,struct folio *folio) { } -static inline void swap_write_unplug (struct swap_iocb *sio) +static inline void swap_write_submit (struct swap_io_ctx *ctx) { }
@@ -425,8 +422,7 @@ static inline void swap_update_readahead(struct folio *folio, { } -static inline int swap_writeout (struct folio *folio, -struct swap_iocb **swap_plug) +static inline int swap_writeout (struct swap_io_ctx *ctx,struct folio *folio) {return 0 ; }
@@ -472,7 +468,7 @@ static inline unsigned int folio_swap_flags(struct folio *folio)#endif /* CONFIG_SWAP */ -int shmem_writeout (struct folio *folio,struct swap_iocb **plug, +int shmem_writeout (struct swap_io_ctx *ctx,struct folio *folio,struct list_head *folio_list);#endif /* _MM_SWAP_H */
mm/swap_state.c +31 -22
135 lines not on this page · raw message
mm/vmscan.c +7 -8
@@ -616,8 +616,8 @@ typedef enum {/* * pageout is called by shrink_folio_list() for each dirty folio. */ -static pageout_t pageout (struct folio *folio,struct address_space *mapping, -struct swap_iocb **plug,struct list_head *folio_list) +static pageout_t pageout (struct swap_io_ctx *ctx,struct address_space *mapping, +struct folio *folio,struct list_head *folio_list) {int res;
@@ -653,9 +653,9 @@ static pageout_t pageout(struct folio *folio, struct address_space *mapping,* the split out folios get added back to folio_list. */ if (shmem_mapping (mapping)) - res =shmem_writeout (folio, plug, folio_list); + res =shmem_writeout (ctx, folio, folio_list);else - res =swap_writeout (folio, plug); + res =swap_writeout (ctx, folio);if (res <0 )handle_write_error (mapping, folio, res);
@@ -1059,7 +1059,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,unsigned int nr_reclaimed =0 , nr_demoted =0 ;unsigned int pgactivate =0 ;bool do_demote_pass; -struct swap_iocb *plug =NULL ; +struct swap_io_ctx ctx = {};folio_batch_init (&free_folios);memset (stat,0 ,sizeof (*stat));
@@ -1387,7 +1387,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,* starts and then write it out here. */ try_to_unmap_flush_dirty (); -switch (pageout (folio, mapping, &plug, folio_list)) { +switch (pageout (&ctx, mapping, folio, folio_list)) {case PAGE_KEEP : goto keep_locked;case PAGE_ACTIVATE :
@@ -1577,8 +1577,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,list_splice (&ret_folios, folio_list);count_vm_events (PGACTIVATE , pgactivate); -if (plug) -swap_write_unplug (plug); +swap_write_submit (&ctx);return nr_reclaimed; }
mm/zswap.c +3 -1
@@ -992,6 +992,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry,struct folio *folio;struct mempolicy *mpol;struct swap_info_struct *si; +struct swap_io_ctx ctx = {};int ret =0 ;/* try to allocate swap cache folio */
@@ -1049,7 +1050,8 @@ static int zswap_writeback_entry(struct zswap_entry *entry,folio_set_reclaim (folio);/* start writeback */ -__swap_writepage (folio,NULL ); +__swap_writepage (&ctx, folio); +swap_write_submit (&ctx); out:if (ret) {
Signature · 1 line
2.53.0
Baoquan He
reviewerRe: [PATCH 2/7] mm/swap: introduce struct swap_io_ctx
Hi Chrisoph,
On 07/13/26 at 11:33am, Christoph Hellwig wrote:
......snip...
mm/madvise.c static int swapin_walk_pmd_entry(pmd_t *pmd, unsigned long start, · line 216
Quoted 8 lines
mm/madvise.c +2
-2@@ -212,15 +212,15 @@ static int swapin_walk_pmd_entry(pmd_t *pmd, unsigned long start,pte_unmap_unlock (ptep, ptl); ptep =NULL ; - folio =read_swap_cache_async (entry,GFP_HIGHUSER_MOVABLE , - vma, addr, &splug); + folio =read_swap_cache_async (&ctx, entry,GFP_HIGHUSER_MOVABLE , + vma, addr);
Could you please help answer the question in v2?
https://lore.kernel.org/all/aiFaWmCWgEWAFj4W@MiWiFi-R3L-srv/T/#u
It's not clear to me why ctx need be put at the 1st parameter. And if only replace &splug with &ctx, there will be one less line of code change, and this applies to other places of change.
...snip...
Re: [PATCH 2/7] mm/swap: introduce struct swap_io_ctx
Quoted 16 lines · Baoquan He
Hi Chrisoph,
On 07/13/26 at 11:33am, Christoph Hellwig wrote:
......snip...mm/madvise.c static int swapin_walk_pmd_entry(pmd_t *pmd, unsigned long start, · line 216
Quoted 8 lines
mm/madvise.c +2
-2@@ -212,15 +212,15 @@ static int swapin_walk_pmd_entry(pmd_t *pmd, unsigned long start,pte_unmap_unlock (ptep, ptl); ptep =NULL ; - folio =read_swap_cache_async (entry,GFP_HIGHUSER_MOVABLE , - vma, addr, &splug); + folio =read_swap_cache_async (&ctx, entry,GFP_HIGHUSER_MOVABLE , + vma, addr);Could you please help answer the question in v2?
https://lore.kernel.org/all/aiFaWmCWgEWAFj4W@MiWiFi-R3L-srv/T/#u
Quoted 3 lines
It's not clear to me why ctx need be put at the 1st parameter. And if only replace &splug with &ctx, there will be one less line of code change, and this applies to other places of change.
It is not needed, but passing a context as the first argument is a common and good convention.
PATCH 3/7 mm/swap: also use struct swap_iocb for block I/O
In reply to Christoph HellwigBlock I/O benefits from batching just as much as remote file systems. Extend struct swap_iocb to support building a bio on the fly as well, and rewrite the block based swap code for it. This especially benefits submit_bio based drivers that do not have the block plugging available, but also saves allocating extra bios for blk-mq drivers.
Add a pre-allocated bio to struct swap_iocb in a union with kiocb used for file system based swap so that struct swap_iocb can be used for all swap I/O, and initialize the pool for it unconditionally.
Various low-level bdev and fs functions are now replaced with a unified can_merge/add/submit scheme.
Note that the block based swap code now uses the same memcg-based check previously added for file system based swap as well.
--- mm/page_io.c | 526 ++++++++++++++++++++++++-------------------------- mm/swap.h | 1 + mm/swapfile.c | 9 +- 3 files changed, 254 insertions(+), 282 deletions(-)
mm/page_io.c +249 -277
@@ -28,54 +28,6 @@#include "swap.h" #include "swap_table.h" -static void __end_swap_bio_write (struct bio *bio) -{ -struct folio *folio =bio_first_folio_all (bio); - -if (bio->bi_status) { -/* -* We failed to write the page out to swap-space. -* Re-dirty the page in order to avoid it being reclaimed. -* Also print a dire warning that things will go BAD (tm) -* very quickly. -* -* Also clear PG_reclaim to avoid folio_rotate_reclaimable() -*/ -folio_mark_dirty (folio); -pr_alert_ratelimited ("Write-error on swap-device (%u:%u:%llu)\n" , -MAJOR (bio_dev (bio)),MINOR (bio_dev (bio)), - (unsigned long long )bio->bi_iter.bi_sector); -folio_clear_reclaim (folio); - } -folio_end_writeback (folio); -} - -static void end_swap_bio_write (struct bio *bio) -{ -__end_swap_bio_write (bio); -bio_put (bio); -} - -static void __end_swap_bio_read (struct bio *bio) -{ -struct folio *folio =bio_first_folio_all (bio); - -if (bio->bi_status) { -pr_alert_ratelimited ("Read-error on swap-device (%u:%u:%llu)\n" , -MAJOR (bio_dev (bio)),MINOR (bio_dev (bio)), - (unsigned long long )bio->bi_iter.bi_sector); - }else { -folio_mark_uptodate (folio); - } -folio_unlock (folio); -} - -static void end_swap_bio_read (struct bio *bio) -{ -__end_swap_bio_read (bio); -bio_put (bio); -} -int generic_swapfile_activate (struct swap_info_struct *sis,struct file *swap_file,sector_t *span)
@@ -316,18 +268,36 @@ static inline void count_swpout_vm_event(struct folio *folio) }#if defined(CONFIG_MEMCG ) && defined(CONFIG_BLK_CGROUP ) +static struct cgroup_subsys_state *folio_memcg_blkg_css (struct folio *folio) +{ +return cgroup_e_css (folio_memcg (folio)->css.cgroup, &io_cgrp_subsys); +} + +static bool folio_blkg_can_merge (struct folio *folio,struct folio *prev_folio) +{ +bool can_merge = true; + +if (folio_memcg_charged (folio) !=folio_memcg_charged (prev_folio)) +return false; +if (folio_memcg_charged (folio)) { +rcu_read_lock (); +if (folio_memcg_blkg_css (folio) != +folio_memcg_blkg_css (prev_folio)) + can_merge = false; +rcu_read_unlock (); + } +return can_merge; +} +static void bio_associate_blkg_from_page (struct bio *bio,struct folio *folio) {struct cgroup_subsys_state *css; -struct mem_cgroup *memcg;if (!folio_memcg_charged (folio))return ; -rcu_read_lock (); - memcg =folio_memcg (folio); - css =cgroup_e_css (memcg->css.cgroup, &io_cgrp_subsys); -if (!css || !css_tryget (css)) + css =folio_memcg_blkg_css (folio); +if (css && !css_tryget (css)) css =NULL ;rcu_read_unlock ();
@@ -336,11 +306,18 @@ static void bio_associate_blkg_from_page(struct bio *bio, struct folio *folio)css_put (css); }#else +static bool folio_blkg_can_merge (struct folio *folio,struct folio *prev_folio) +{ +return true; +}#define bio_associate_blkg_from_page (bio, folio) do { } while (0)#endif /* CONFIG_MEMCG && CONFIG_BLK_CGROUP */ struct swap_iocb { -struct kiocb iocb; +union { +struct kiocb iocb; +struct bio bio; + };struct bio_vec bvecs[SWAP_CLUSTER_MAX ];int nr_bvecs;int len;
@@ -360,171 +337,70 @@ int sio_pool_init(void)return 0 ; } -static void sio_write_complete (struct kiocb *iocb,long ret) +static bool swap_can_merge (struct swap_io_ctx *ctx,struct folio *folio, +int rw) { - struct swap_iocb *sio = container_of(iocb, struct swap_iocb, iocb); -struct page *page = sio->bvecs[0 ].bv_page; -int p; +struct swap_info_struct *sis =__swap_entry_to_info (folio->swap); +struct bio_vec *last_bv = &ctx->sio->bvecs[ctx->sio->nr_bvecs -1 ]; +struct folio *prev_folio =bvec_folio (last_bv); +size_t prev_folio_size =folio_size (prev_folio); -if (ret != sio->len) { -/* -* In the case of swap-over-nfs, this can be a -* temporary failure if the system has limited -* memory for allocating transmit buffers. -* Mark the page dirty and avoid -* folio_rotate_reclaimable but rate-limit the -* messages. -*/ -pr_err_ratelimited ("Write error %ld on dio swapfile (%llu)\n" , - ret,swap_dev_pos (page_swap_entry (page))); -for (p =0 ; p < sio->nr_bvecs; p++) { - page = sio->bvecs[p].bv_page; -set_page_dirty (page); -ClearPageReclaim (page); - } - } +if (ctx->sis != sis) +return false; -for (p =0 ; p < sio->nr_bvecs; p++) -end_page_writeback (sio->bvecs[p].bv_page); +if (sis->flags &SWP_FS_OPS ) { +if (swap_dev_pos (folio->swap) != +swap_dev_pos (prev_folio->swap) + prev_folio_size) +return false; + }else { +if (swap_folio_sector (folio) != +swap_folio_sector (prev_folio) + + (prev_folio_size >>SECTOR_SHIFT )) +return false; +if (rw ==WRITE && !folio_blkg_can_merge (folio, prev_folio)) +return false; + } -mempool_free (sio, sio_pool); +return true; } -static void swap_writepage_fs (struct swap_io_ctx *ctx,struct folio *folio) +static void swap_add_folio (struct swap_io_ctx *ctx,struct folio *folio,int rw) { -struct swap_iocb *sio = ctx->sio;struct swap_info_struct *sis =__swap_entry_to_info (folio->swap); -struct file *swap_file = sis->swap_file; -loff_t pos =swap_dev_pos (folio->swap); +struct swap_iocb *sio = ctx->sio; -count_swpout_vm_event (folio); -folio_start_writeback (folio); -folio_unlock (folio); -if (sio) { -if (sio->iocb.ki_filp != swap_file || - sio->iocb.ki_pos + sio->len != pos) { +if (sio && !swap_can_merge (ctx, folio, rw)) { +if (rw ==WRITE )swap_write_submit (ctx); - sio =NULL ; - } +else +swap_read_submit (ctx); + sio = ctx->sio; } +if (!sio) { - sio =mempool_alloc (sio_pool,GFP_NOIO ); -init_sync_kiocb (&sio->iocb, swap_file); - sio->iocb.ki_complete = sio_write_complete; - sio->iocb.ki_pos = pos; + ctx->sis = sis; + ctx->sio = sio =mempool_alloc (sio_pool,GFP_NOIO ); sio->nr_bvecs =0 ; sio->len =0 ; }bvec_set_folio (&sio->bvecs[sio->nr_bvecs], folio,folio_size (folio),0 ); sio->len +=folio_size (folio); - sio->nr_bvecs +=1 ; -if (sio->nr_bvecs ==ARRAY_SIZE (sio->bvecs)) { -swap_write_submit (ctx); - sio =NULL ; +if (++sio->nr_bvecs ==ARRAY_SIZE (sio->bvecs)) { +if (rw ==WRITE ) +swap_write_submit (ctx); +else +swap_read_submit (ctx); } - ctx->sio = sio; } -static void swap_writepage_bdev_sync (struct folio *folio, -struct swap_info_struct *sis) -{ -struct bio_vec bv; -struct bio bio; - -bio_init (&bio, sis->bdev, &bv,1 ,REQ_OP_WRITE |REQ_SWAP ); - bio.bi_iter.bi_sector =swap_folio_sector (folio); -bio_add_folio_nofail (&bio, folio,folio_size (folio),0 ); - -bio_associate_blkg_from_page (&bio, folio); -count_swpout_vm_event (folio); - -folio_start_writeback (folio); -folio_unlock (folio); - -submit_bio_wait (&bio); -__end_swap_bio_write (&bio); -} - -static void swap_writepage_bdev_async (struct folio *folio, -struct swap_info_struct *sis) +void __swap_writepage (struct swap_io_ctx *ctx,struct folio *folio) { -struct bio *bio; - - bio =bio_alloc (sis->bdev,1 ,REQ_OP_WRITE |REQ_SWAP ,GFP_NOIO ); - bio->bi_iter.bi_sector =swap_folio_sector (folio); - bio->bi_end_io = end_swap_bio_write; -bio_add_folio_nofail (bio, folio,folio_size (folio),0 ); +VM_BUG_ON_FOLIO (!folio_test_swapcache (folio), folio); -bio_associate_blkg_from_page (bio, folio);count_swpout_vm_event (folio);folio_start_writeback (folio);folio_unlock (folio); -submit_bio (bio); -} - -void __swap_writepage (struct swap_io_ctx *ctx,struct folio *folio) -{ -struct swap_info_struct *sis =__swap_entry_to_info (folio->swap); - -VM_BUG_ON_FOLIO (!folio_test_swapcache (folio), folio); -/* -* ->flags can be updated non-atomically, -* but that will never affect SWP_FS_OPS, so the data_race -* is safe. -*/ -if (data_race (sis->flags &SWP_FS_OPS )) -swap_writepage_fs (ctx, folio); -/* -* ->flags can be updated non-atomically, -* but that will never affect SWP_SYNCHRONOUS_IO, so the data_race -* is safe. -*/ -else if (data_race (sis->flags &SWP_SYNCHRONOUS_IO )) -swap_writepage_bdev_sync (folio, sis); -else -swap_writepage_bdev_async (folio, sis); -} - -void swap_write_submit (struct swap_io_ctx *ctx) -{ -struct swap_iocb *sio = ctx->sio; -struct iov_iter from; -int ret; - -if (!sio) -return ; - -iov_iter_bvec (&from,ITER_SOURCE , sio->bvecs, sio->nr_bvecs, sio->len); - ret = sio->iocb.ki_filp->f_mapping->a_ops->swap_rw (&sio->iocb, &from); -if (ret != -EIOCBQUEUED ) -sio_write_complete (&sio->iocb, ret); - ctx->sio =NULL ; -} - -static void sio_read_complete (struct kiocb *iocb,long ret) -{ - struct swap_iocb *sio = container_of(iocb, struct swap_iocb, iocb); -int p; - -if (ret == sio->len) { -for (p =0 ; p < sio->nr_bvecs; p++) { -struct folio *folio =bvec_folio (&sio->bvecs[p]); - -count_mthp_stat (folio_order (folio),MTHP_STAT_SWPIN ); -count_memcg_folio_events (folio,PSWPIN ,folio_nr_pages (folio)); -folio_mark_uptodate (folio); -folio_unlock (folio); - } -count_vm_events (PSWPIN , sio->len >>PAGE_SHIFT ); - }else { -for (p =0 ; p < sio->nr_bvecs; p++) { -struct folio *folio =bvec_folio (&sio->bvecs[p]); - -folio_unlock (folio); - } -pr_alert_ratelimited ("Read-error on swap-device\n" ); - } -mempool_free (sio, sio_pool); +swap_add_folio (ctx, folio,WRITE ); }/*
@@ -590,74 +466,6 @@ static bool swap_read_folio_zeromap(struct folio *folio)return true; } -static void swap_read_folio_fs (struct swap_io_ctx *ctx,struct folio *folio) -{ -struct swap_info_struct *sis =__swap_entry_to_info (folio->swap); -struct swap_iocb *sio = ctx->sio; -loff_t pos =swap_dev_pos (folio->swap); - -if (sio) { -if (sio->iocb.ki_filp != sis->swap_file || - sio->iocb.ki_pos + sio->len != pos) { -swap_read_submit (ctx); - sio =NULL ; - } - } -if (!sio) { - sio =mempool_alloc (sio_pool,GFP_KERNEL ); -init_sync_kiocb (&sio->iocb, sis->swap_file); - sio->iocb.ki_pos = pos; - sio->iocb.ki_complete = sio_read_complete; - sio->nr_bvecs =0 ; - sio->len =0 ; - } -bvec_set_folio (&sio->bvecs[sio->nr_bvecs], folio,folio_size (folio),0 ); - sio->len +=folio_size (folio); - sio->nr_bvecs +=1 ; -if (sio->nr_bvecs ==ARRAY_SIZE (sio->bvecs)) { -swap_read_submit (ctx); - sio =NULL ; - } - ctx->sio = sio; -} - -static void swap_read_folio_bdev_sync (struct folio *folio, -struct swap_info_struct *sis) -{ -struct bio_vec bv; -struct bio bio; - -bio_init (&bio, sis->bdev, &bv,1 ,REQ_OP_READ ); - bio.bi_iter.bi_sector =swap_folio_sector (folio); -bio_add_folio_nofail (&bio, folio,folio_size (folio),0 ); -/* -* Keep this task valid during swap readpage because the oom killer may -* attempt to access it in the page fault retry time check. -*/ -get_task_struct (current); -count_mthp_stat (folio_order (folio),MTHP_STAT_SWPIN ); -count_memcg_folio_events (folio,PSWPIN ,folio_nr_pages (folio)); -count_vm_events (PSWPIN ,folio_nr_pages (folio)); -submit_bio_wait (&bio); -__end_swap_bio_read (&bio); -put_task_struct (current); -} - -static void swap_read_folio_bdev_async (struct folio *folio, -struct swap_info_struct *sis) -{ -struct bio *bio; - - bio =bio_alloc (sis->bdev,1 ,REQ_OP_READ ,GFP_KERNEL ); - bio->bi_iter.bi_sector =swap_folio_sector (folio); - bio->bi_end_io = end_swap_bio_read; -bio_add_folio_nofail (bio, folio,folio_size (folio),0 ); -count_mthp_stat (folio_order (folio),MTHP_STAT_SWPIN ); -count_memcg_folio_events (folio,PSWPIN ,folio_nr_pages (folio)); -count_vm_events (PSWPIN ,folio_nr_pages (folio)); -submit_bio (bio); -} -void swap_read_folio (struct swap_io_ctx *ctx,struct folio *folio) {struct swap_info_struct *sis =__swap_entry_to_info (folio->swap);
@@ -691,14 +499,7 @@ void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio)/* We have to read from slower devices. Increase zswap protection. */ zswap_folio_swapin (folio); - -if (data_race (sis->flags &SWP_FS_OPS )) { -swap_read_folio_fs (ctx, folio); - }else if (synchronous) { -swap_read_folio_bdev_sync (folio, sis); - }else { -swap_read_folio_bdev_async (folio, sis); - } +swap_add_folio (ctx, folio,READ ); finish:if (workingset) {
@@ -708,18 +509,189 @@ void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio)delayacct_swapin_end (); } -void swap_read_submit (struct swap_io_ctx *ctx) +static void swap_write_end (struct swap_iocb *sio,bool failed) +{ +int p; + +for (p =0 ; p < sio->nr_bvecs; p++) { +struct page *page = sio->bvecs[p].bv_page; + +if (failed) { +set_page_dirty (page); +ClearPageReclaim (page); + } +end_page_writeback (page); + } +mempool_free (sio, sio_pool); +} + +static void swap_fs_write_complete (struct kiocb *iocb,long ret) +{ + struct swap_iocb *sio = container_of(iocb, struct swap_iocb, iocb); +bool failed = ret != sio->len; + +if (failed) { +struct page *page = sio->bvecs[0 ].bv_page; + +/* +* In the case of swap-over-nfs, this can be a temporary failure +* if the system has limited memory for allocating transmit +* buffers. Mark the page dirty and avoid +* folio_rotate_reclaimable but rate-limit the messages. +*/ +pr_err_ratelimited ("Write error %ld on dio swapfile (%llu)\n" , + ret,swap_dev_pos (page_swap_entry (page))); + } + +swap_write_end (sio, failed); +} + +static void end_swap_bio_write (struct bio *bio) +{ + struct swap_iocb *sio = container_of(bio, struct swap_iocb, bio); +bool failed = !!bio->bi_status; + +if (failed) +pr_alert_ratelimited ("Write-error on swap-device (%u:%u:%llu)\n" , +MAJOR (bio_dev (bio)),MINOR (bio_dev (bio)), + (unsigned long long )bio->bi_iter.bi_sector); +bio_uninit (bio); +swap_write_end (sio, failed); +} + +static void swap_read_end (struct swap_iocb *sio,bool failed) +{ +int p; + +for (p =0 ; p < sio->nr_bvecs; p++) { +struct folio *folio =bvec_folio (&sio->bvecs[p]); + +if (!failed) { +count_mthp_stat (folio_order (folio),MTHP_STAT_SWPIN ); +count_memcg_folio_events (folio,PSWPIN , +folio_nr_pages (folio)); +folio_mark_uptodate (folio); + } +folio_unlock (folio); + } + +if (!failed) +count_vm_events (PSWPIN , sio->len >>PAGE_SHIFT ); + +mempool_free (sio, sio_pool); +} + +static void swap_fs_read_complete (struct kiocb *iocb,long ret) +{ + struct swap_iocb *sio = container_of(iocb, struct swap_iocb, iocb); +bool failed = ret != sio->len; + +if (failed) +pr_alert_ratelimited ("Read-error on swap-device\n" ); +swap_read_end (sio, failed); +} + +static void swap_bio_read_end_io (struct bio *bio) +{ + struct swap_iocb *sio = container_of(bio, struct swap_iocb, bio); +bool failed = !!bio->bi_status; + +if (failed) +pr_alert_ratelimited ("Read-error on swap-device (%u:%u:%llu)\n" , +MAJOR (bio_dev (bio)),MINOR (bio_dev (bio)), + (unsigned long long )bio->bi_iter.bi_sector); +bio_uninit (bio); +swap_read_end (sio, failed); +} + +static void swap_bdev_submit_write (struct swap_io_ctx *ctx) {struct swap_iocb *sio = ctx->sio; -struct iov_iter from; +struct bio *bio = &sio->bio; + +bio_init (bio, ctx->sis->bdev, sio->bvecs,ARRAY_SIZE (sio->bvecs), +REQ_OP_WRITE |REQ_SWAP ); + bio->bi_iter.bi_size = sio->len; + bio->bi_iter.bi_sector =swap_folio_sector (bio_first_folio_all (bio)); +bio_associate_blkg_from_page (bio,bio_first_folio_all (bio)); + +if (ctx->sis->flags &SWP_SYNCHRONOUS_IO ) { +submit_bio_wait (bio); +end_swap_bio_write (bio); + }else { + bio->bi_end_io = end_swap_bio_write; +submit_bio (bio); + } +} + +static void swap_bdev_submit_read (struct swap_io_ctx *ctx) +{ +struct swap_iocb *sio = ctx->sio; +struct bio *bio = &sio->bio; + +bio_init (bio, ctx->sis->bdev, sio->bvecs,ARRAY_SIZE (sio->bvecs), +REQ_OP_READ ); + bio->bi_iter.bi_size = sio->len; + bio->bi_iter.bi_sector =swap_folio_sector (bio_first_folio_all (bio)); + +if (ctx->sis->flags &SWP_SYNCHRONOUS_IO ) { +/* +* Keep this task valid during swap readpage because the oom +* killer may attempt to access it in the page fault retry +* time check. +*/ +get_task_struct (current); +submit_bio_wait (bio); +swap_bio_read_end_io (bio); +put_task_struct (current); + }else { + bio->bi_end_io = swap_bio_read_end_io; +submit_bio (bio); + } +} + +static void swap_fs_submit (struct swap_io_ctx *ctx,int rw) +{ +struct swap_iocb *sio = ctx->sio; +struct iov_iter iter;int ret; -if (!sio) -return ; +init_sync_kiocb (&sio->iocb, ctx->sis->swap_file); + sio->iocb.ki_pos =swap_dev_pos (bvec_folio (&sio->bvecs[0 ])->swap); +if (rw ==WRITE ) + sio->iocb.ki_complete = swap_fs_write_complete; +else + sio->iocb.ki_complete = swap_fs_read_complete; -iov_iter_bvec (&from,ITER_DEST , sio->bvecs, sio->nr_bvecs, sio->len); - ret = sio->iocb.ki_filp->f_mapping->a_ops->swap_rw (&sio->iocb, &from); +iov_iter_bvec (&iter, rw ==WRITE ?ITER_SOURCE :ITER_DEST , + sio->bvecs, sio->nr_bvecs, sio->len); + ret = sio->iocb.ki_filp->f_mapping->a_ops->swap_rw (&sio->iocb, &iter);if (ret != -EIOCBQUEUED ) -sio_read_complete (&sio->iocb, ret); + sio->iocb.ki_complete (&sio->iocb, ret); +} + +void swap_write_submit (struct swap_io_ctx *ctx) +{ +if (!ctx->sio) +return ; + +if (ctx->sis->flags &SWP_FS_OPS ) +swap_fs_submit (ctx,WRITE ); +else +swap_bdev_submit_write (ctx); + ctx->sio =NULL ; + ctx->sis =NULL ; +} + +void swap_read_submit (struct swap_io_ctx *ctx) +{ +if (!ctx->sio) +return ; + +if (ctx->sis->flags &SWP_FS_OPS ) +swap_fs_submit (ctx,READ ); +else +swap_bdev_submit_read (ctx); ctx->sio =NULL ; + ctx->sis =NULL ; }
mm/swap.h +1 -0
@@ -79,6 +79,7 @@ enum swap_cluster_flags {struct swap_io_ctx {struct swap_iocb *sio; +struct swap_info_struct *sis; };#ifdef CONFIG_SWAP
mm/swapfile.c +4 -5
@@ -2946,6 +2946,10 @@ static int setup_swap_extents(struct swap_info_struct *sis,struct inode *inode = mapping->host;int ret; + ret =sio_pool_init (); +if (ret) +return ret; +if (S_ISBLK (inode->i_mode)) { ret =add_swap_extent (sis,0 , sis->max,0 ); *span = sis->pages;
@@ -2957,11 +2961,6 @@ static int setup_swap_extents(struct swap_info_struct *sis,if (ret <0 )return ret; sis->flags |=SWP_ACTIVATED ; -if ((sis->flags &SWP_FS_OPS ) && -sio_pool_init () !=0 ) { -destroy_swap_extents (sis, swap_file); -return -ENOMEM ; - }return ret; }
Signature · 1 line
2.53.0
Re: [PATCH 3/7] mm/swap: also use struct swap_iocb for block I/O
Quoted 16 lines · Christoph Hellwig
Block I/O benefits from batching just as much as remote file systems. Extend struct
swap_iocbto support building a bio on the fly as well, and rewrite the block based swap code for it. This especially benefitssubmit_biobased drivers that do not have the block plugging available, but also saves allocating extra bios for blk-mq drivers.Add a pre-allocated bio to struct
swap_iocbin a union with kiocb used for file system based swap so that structswap_iocbcan be used for all swap I/O, and initialize the pool for it unconditionally.Various low-level bdev and fs functions are now replaced with a unified can_merge/add/submit scheme.
Note that the block based swap code now uses the same memcg-based check previously added for file system based swap as well.
Quoted 6 lines
Signed-off-by Christoph Hellwig <[email protected]>--- mm/page_io.c | 526 ++++++++++++++++++++++++-------------------------- mm/swap.h | 1 + mm/swapfile.c | 9 +- 3 files changed, 254 insertions(+), 282 deletions(-)
Hi, thanks for the great work, sorry for the joining this late as I spent quite some time on the new swap priority list, now checking this series.
Quoted 9 lines
mm/page_io.c +0
-0@@ -28,54 +28,6 @@#include "swap.h" #include "swap_table.h"
...
Quoted 21 lines
mm/page_io.c +8
-1@@ -336,11 +306,18 @@ static void bio_associate_blkg_from_page(struct bio *bio, struct folio *folio)css_put (css); }#else +static bool folio_blkg_can_merge (struct folio *folio,struct folio *prev_folio) +{ +return true; +}#define bio_associate_blkg_from_page (bio, folio) do { } while (0)#endif /* CONFIG_MEMCG && CONFIG_BLK_CGROUP */ struct swap_iocb { -struct kiocb iocb; +union { +struct kiocb iocb; +struct bio bio; + };struct bio_vec bvecs[SWAP_CLUSTER_MAX ];int nr_bvecs;int len;
mm/page_io.c int sio_pool_init(void) · line 341
Quoted 6 lines
mm/page_io.c +2
-1@@ -360,171 +337,70 @@ int sio_pool_init(void)return 0 ; } -static void sio_write_complete (struct kiocb *iocb,long ret) +static bool swap_can_merge (struct swap_io_ctx *ctx,struct folio *folio, +int rw)
Will a bool rw suits better?
Quoted 48 lines
mm/page_io.c +19
-23{ - struct swap_iocb *sio = container_of(iocb, struct swap_iocb, iocb); -struct page *page = sio->bvecs[0 ].bv_page; -int p; +struct swap_info_struct *sis =__swap_entry_to_info (folio->swap); +struct bio_vec *last_bv = &ctx->sio->bvecs[ctx->sio->nr_bvecs -1 ]; +struct folio *prev_folio =bvec_folio (last_bv); +size_t prev_folio_size =folio_size (prev_folio); -if (ret != sio->len) { -/* -* In the case of swap-over-nfs, this can be a -* temporary failure if the system has limited -* memory for allocating transmit buffers. -* Mark the page dirty and avoid -* folio_rotate_reclaimable but rate-limit the -* messages. -*/ -pr_err_ratelimited ("Write error %ld on dio swapfile (%llu)\n" , - ret,swap_dev_pos (page_swap_entry (page))); -for (p =0 ; p < sio->nr_bvecs; p++) { - page = sio->bvecs[p].bv_page; -set_page_dirty (page); -ClearPageReclaim (page); - } - } +if (ctx->sis != sis) +return false; -for (p =0 ; p < sio->nr_bvecs; p++) -end_page_writeback (sio->bvecs[p].bv_page); +if (sis->flags &SWP_FS_OPS ) { +if (swap_dev_pos (folio->swap) != +swap_dev_pos (prev_folio->swap) + prev_folio_size) +return false; + }else { +if (swap_folio_sector (folio) != +swap_folio_sector (prev_folio) + + (prev_folio_size >>SECTOR_SHIFT )) +return false; +if (rw ==WRITE && !folio_blkg_can_merge (folio, prev_folio)) +return false; + } -mempool_free (sio, sio_pool); +return true; }
mm/page_io.c int sio_pool_init(void) · line 436
Quoted 10 lines
mm/page_io.c +0
-9-static void swap_writepage_bdev_sync (struct folio *folio, -struct swap_info_struct *sis) -{ -struct bio_vec bv; -struct bio bio; - -bio_init (&bio, sis->bdev, &bv,1 ,REQ_OP_WRITE |REQ_SWAP ); - bio.bi_iter.bi_sector =swap_folio_sector (folio); -bio_add_folio_nofail (&bio, folio,folio_size (folio),0 );
bio_add_folio_nofail also increased bi_vcnt, now it's open coded and bi_vcnt stays zero, it's not used right now, could there be any user of it in the future?
Quoted 5 lines
mm/page_io.c +0
-5- -bio_associate_blkg_from_page (&bio, folio); -count_swpout_vm_event (folio); - -folio_start_writeback (folio);
mm/page_io.c int sio_pool_init(void) · line 397
Quoted 10 lines
mm/page_io.c +1
-8-folio_unlock (folio); - -submit_bio_wait (&bio); -__end_swap_bio_write (&bio); -} - -static void swap_writepage_bdev_async (struct folio *folio, -struct swap_info_struct *sis) +void __swap_writepage (struct swap_io_ctx *ctx,struct folio *folio) {
... The git diff is not very human readable but fine, here we have __swap_writepage not doing IO anymore but add the folio and deferred the IO ...
mm/page_io.c int sio_pool_init(void) · line 403
Quoted 1 line
mm/page_io.c +1
-0+swap_add_folio (ctx, folio,WRITE );
So this removes swap_writepage_bdev_sync (which was calling submit_bio_wait), we are replying on a later swap_write_submit, and just add the folio to ctx.
In the previous patch, swap_write_submit is called after the whole batch of folio is iterated, so there is no more IO completion in pageout().
Before the series, pageout will detect for writeback finished and clean pages under PAGE_SUCCESS, if the IO of a page is done and writeback flag cleaned, the page get released immediately. But now, almost all pages will be put back to the LRU waiting for rotation. This also make the reclaim have a stronger bias towards reclaim files, and anon reclaim become less effective.
I do observe a real performance regression with classical LRU and ZRAM, ordinary swap seems also very slightly effected but could be noise, test is done on my laptop by building the kernel using make -j12 and tinyconfig in a 2G VM with mild pressure:
Before this series (With NVME, in a virtual machine, NVME drive exposed to the VM by virtio):
152.02user 42.34system 0:32.07elapsed
3303408inputs+0outputs (83567major+15910796minor)pagefaults
After this series:
155.72user 46.19system 0:33.88elapsed 3617720inputs+0outputs (94045major+16094079minor)pagefaults
Before this series (with 8G ZRAM only):
175.39user 125.89system 0:40.54elapsed 26581840inputs+0outputs (1186391major+22698964minor)pagefaults
After this sereis:
165.59user 216.02system 1:04.06elapsed <- the problemic one. 53531464inputs+0outputs (2550441major+31974007minor)pagefaults
Because all anon folios will be derfer freed on next iteration of the LRU, which caused a much higher memory pressure.
And, very interestingly, with MGLRU, things are looking better in every way (which is kind of expected, see below):
Before this series (With NVME, in a virtual machine, NVME drive exposed to the VM by virtio):
155.96user 42.15system 0:30.57elapsed 2057792inputs+0outputs (75116major+15860836minor)pagefaults
After this series:
150.42user 44.18system 0:31.21elapsed 2545616inputs+0outputs (69325major+16141325minor)pagefaults
Before this series (with 8G ZRAM only):
175.57user 120.27system 0:36.32elapsed 24655320inputs+0outputs (1568694major+22468687minor)pagefaults
After:
180.59user 119.43system 0:36.18elapsed 23893384inputs+0outputs (1546158major+22302635minor)pagefaults
I only testes each case for 5 times and take the median result, it could be a bit noisy but good enough for now, the results are mostly stable.
So in summary when using classical LRU with ZRAM, there is a major regression because pageout can no longer free any page directly (40s -> 1min), all page have to do a round trip to the LRU again before gets freed. Other cases are mostly fine, but maybe pageout might catch a few clean folios before even for ordinary devices and free them directly? So we are seeming a slightly higher major fault due to deferred free / IO? (Could be noise though).
Things are looking good with MGLRU, because MGLRU have a secondary retry for clean folio direct reclaim even if pageout() didn't do that (see the "retry folios that may have missed folio_rotate_reclaimable" comment, that comment isn't that correct either though).
I remember Barry mentioned that someone once tried to implement a similar secondary retry for classical LRU too, but got rejected.
I think this better be fixed form the LRU side to always try batch free the clean folios after one iteration, which is also part of unifying MGLRU / LRU, compression (either ZRAM / ZSWAP) may also benefit from batching though, and free the folios in batch it could be even better.
That could take more time though, so shall we have a way to just submit the IO early for now?
Re: [PATCH 3/7] mm/swap: also use struct swap_iocb for block I/O
Quoted 5 lines · Kairui Song
mm/page_io.c int sio_pool_init(void) · line 341
Quoted 3 lines
mm/page_io.c +2
-1-static void sio_write_complete (struct kiocb *iocb,long ret) +static bool swap_can_merge (struct swap_io_ctx *ctx,struct folio *folio, +int rw)Will a bool rw suits better?
Not really. Bools as arguents always have the downside that it is really hard to guess what they mean based on the callsite.
While READ/WRITE really should be enum, it's probably the best we have right now.
Quoted 5 lines
Review
Quoted 1 line
(quoted diff) +0
-1- bio_add_folio_nofail(&bio, folio, folio_size(folio), 0);
bio_add_folio_nofailalso increasedbi_vcnt, now it's open coded andbi_vcntstays zero, it's not used right now, could there be any user of it in the future?
bi_vcnt only exists to assist bio_add_page*/bio_add_folio*. So if we assign the bio_vecs manually we don't need it. If we ever switch to using bio_add_folio_nofail we'd need it, but then we'd already maintain it in there :)
Quoted 2 lines
So this removes
swap_writepage_bdev_sync(which was callingsubmit_bio_wait), we are replying on a laterswap_write_submit, and just add the folio to ctx.
For SWP_SYNCHRONOUS_IO, yes.
Quoted 16 lines
Before the series, pageout will detect for writeback finished and clean pages under
PAGE_SUCCESS, if the IO of a page is done and writeback flag cleaned, the page get released immediately. But now, almost all pages will be put back to the LRU waiting for rotation. This also make the reclaim have a stronger bias towards reclaim files, and anon reclaim become less effective.I do observe a real performance regression with classical LRU and ZRAM, ordinary swap seems also very slightly effected but could be noise, test is done on my laptop by building the kernel using make -j12 and tinyconfig in a 2G VM with mild pressure:
Before this series (with 8G ZRAM only):
175.39user 125.89system 0:40.54elapsed 26581840inputs+0outputs (1186391major+22698964minor)pagefaults
Quoted 6 lines
After this sereis:
165.59user 216.02system 1:04.06elapsed <- the problemic one. 53531464inputs+0outputs (2550441major+31974007minor)pagefaultsBecause all anon folios will be derfer freed on next iteration of the LRU, which caused a much higher memory pressure.
So this is zswap which does set SWP_SYNCHRONOUS_IO. I guess forcing a submit after each bio_vec for SWP_SYNCHRONOUS_IO might make some sense as a workaround, although it is a bit ugly.
Re: [PATCH 3/7] mm/swap: also use struct swap_iocb for block I/O
Quoted 50 lines · Christoph Hellwig
Quoted 5 lines · Kairui Song
mm/page_io.c int sio_pool_init(void) · line 341
Quoted 3 lines
mm/page_io.c +2
-1-static void sio_write_complete (struct kiocb *iocb,long ret) +static bool swap_can_merge (struct swap_io_ctx *ctx,struct folio *folio, +int rw)Will a bool rw suits better?
Not really. Bools as arguents always have the downside that it is really hard to guess what they mean based on the callsite.
While READ/WRITE really should be enum, it's probably the best we have right now.
Quoted 5 lines
Quoted 1 line
-
bio_add_folio_nofail(&bio, folio, folio_size(folio), 0);
bio_add_folio_nofailalso increasedbi_vcnt, now it's open coded andbi_vcntstays zero, it's not used right now, could there be any user of it in the future?
bi_vcntonly exists to assist bio_add_page*/bio_add_folio*. So if we assign thebio_vecsmanually we don't need it. If we ever switch to usingbio_add_folio_nofailwe'd need it, but then we'd already maintain it in there :)Quoted 2 lines
So this removes
swap_writepage_bdev_sync(which was callingsubmit_bio_wait), we are replying on a laterswap_write_submit, and just add the folio to ctx.For
SWP_SYNCHRONOUS_IO, yes.Quoted 20 lines
Before the series, pageout will detect for writeback finished and clean pages under
PAGE_SUCCESS, if the IO of a page is done and writeback flag cleaned, the page get released immediately. But now, almost all pages will be put back to the LRU waiting for rotation. This also make the reclaim have a stronger bias towards reclaim files, and anon reclaim become less effective.I do observe a real performance regression with classical LRU and ZRAM, ordinary swap seems also very slightly effected but could be noise, test is done on my laptop by building the kernel using make -j12 and tinyconfig in a 2G VM with mild pressure:
Before this series (with 8G ZRAM only):
175.39user 125.89system 0:40.54elapsed 26581840inputs+0outputs (1186391major+22698964minor)pagefaultsAfter this sereis:
165.59user 216.02system 1:04.06elapsed <- the problemic one. 53531464inputs+0outputs (2550441major+31974007minor)pagefaults
Quoted 6 lines
Quoted 2 lines
Because all anon folios will be derfer freed on next iteration of the LRU, which caused a much higher memory pressure.
So this is zswap which does set
SWP_SYNCHRONOUS_IO. I guess forcing a submit after eachbio_vecforSWP_SYNCHRONOUS_IOmight make some sense as a workaround, although it is a bit ugly.
Thanks, a SWP_SYNCHRONOUS_IO check here sounds good to me. This check can be dropped if we also drop the writeback check in pageout by letting LRU / MGLRU have a unified batch free after pageout, things would be pretty by then I think.
Re: [PATCH 3/7] mm/swap: also use struct swap_iocb for block I/O
Quoted 54 lines · Kairui Song
Quoted 53 lines · Christoph Hellwig
Quoted 5 lines · Kairui Song
Review
Quoted 3 lines
(quoted diff) +2
-1-static void sio_write_complete(struct kiocb *iocb, long ret) +static bool swap_can_merge(struct swap_io_ctx *ctx, struct folio *folio, + int rw)Will a bool rw suits better?
Not really. Bools as arguents always have the downside that it is really hard to guess what they mean based on the callsite.
While READ/WRITE really should be enum, it's probably the best we have right now.
Quoted 5 lines
Quoted 1 line
-
bio_add_folio_nofail(&bio, folio, folio_size(folio), 0);
bio_add_folio_nofailalso increasedbi_vcnt, now it's open coded andbi_vcntstays zero, it's not used right now, could there be any user of it in the future?
bi_vcntonly exists to assist bio_add_page*/bio_add_folio*. So if we assign thebio_vecsmanually we don't need it. If we ever switch to usingbio_add_folio_nofailwe'd need it, but then we'd already maintain it in there :)Quoted 2 lines
So this removes
swap_writepage_bdev_sync(which was callingsubmit_bio_wait), we are replying on a laterswap_write_submit, and just add the folio to ctx.For
SWP_SYNCHRONOUS_IO, yes.Quoted 22 lines
Before the series, pageout will detect for writeback finished and clean pages under
PAGE_SUCCESS, if the IO of a page is done and writeback flag cleaned, the page get released immediately. But now, almost all pages will be put back to the LRU waiting for rotation. This also make the reclaim have a stronger bias towards reclaim files, and anon reclaim become less effective.I do observe a real performance regression with classical LRU and ZRAM, ordinary swap seems also very slightly effected but could be noise, test is done on my laptop by building the kernel using make -j12 and tinyconfig in a 2G VM with mild pressure:
Before this series (with 8G ZRAM only):
175.39user 125.89system 0:40.54elapsed 26581840inputs+0outputs (1186391major+22698964minor)pagefaultsAfter this sereis:
165.59user 216.02system 1:04.06elapsed <- the problemic one. 53531464inputs+0outputs (2550441major+31974007minor)pagefaultsBecause all anon folios will be derfer freed on next iteration of the LRU, which caused a much higher memory pressure.
Quoted 8 lines
Quoted 3 lines
So this is zswap which does set
SWP_SYNCHRONOUS_IO. I guess forcing a submit after eachbio_vecforSWP_SYNCHRONOUS_IOmight make some sense as a workaround, although it is a bit ugly.Thanks, a
SWP_SYNCHRONOUS_IOcheck here sounds good to me. This check can be dropped if we also drop the writeback check in pageout by letting LRU / MGLRU have a unified batch free after pageout, things would be pretty by then I think.
Does this look good to you? It has passed some basic zram sanity checking here:
---
From 31c1466eacefb68d41848cfbb5dcd4f86073256a Mon Sep 17 00:00:00 2001
From: Christoph Hellwig <[email protected]>
Date: Tue, 21 Jul 2026 06:42:26 +0200
Subject: mm/swap: revert to single-folio writes for synchronous swap devices
Kairui Song reported that zram benefits from submitting each folio directly instead of batching up I/O because the classic LRU scanning benefits from clearing the folio writeback bit in the scan loop.
Accommodate that by kicking off reads for synchronous devices for each iteration.
--- mm/page_io.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
mm/page_io.c +10 -1
@@ -358,7 +358,16 @@ static void swap_add_folio(struct swap_io_ctx *ctx, struct folio *folio, int rw) }bvec_set_folio (&sio->bvecs[sio->nr_bvecs], folio,folio_size (folio),0 ); sio->len +=folio_size (folio); -if (++sio->nr_bvecs ==ARRAY_SIZE (sio->bvecs)) { + +/* +* Write out the iocb if we filled it, or if the device is synchronous. +* +* The latter is to work around expectations in the classic LRU code +* which make synchronous clearing of the folio writeback flag in the +* reclaim path beneficial. +*/ +if (++sio->nr_bvecs ==ARRAY_SIZE (sio->bvecs) || + (rw ==WRITE && (sis->flags &SWP_SYNCHRONOUS_IO ))) {if (rw ==WRITE )swap_write_submit (ctx);else
Signature · 1 line
2.53.0
Re: [PATCH 3/7] mm/swap: also use struct swap_iocb for block I/O
Quoted 45 lines · Christoph Hellwig
Does this look good to you? It has passed some basic zram sanity checking here:
---
From31c1466eacefb68d41848cfbb5dcd4f86073256aMon Sep 17 00:00:00 2001
From: Christoph Hellwig <[email protected]>
Date: Tue, 21 Jul 2026 06:42:26 +0200
Subject: mm/swap: revert to single-folio writes for synchronous swap devicesKairui Song reported that zram benefits from submitting each folio directly instead of batching up I/O because the classic LRU scanning benefits from clearing the folio writeback bit in the scan loop.
Accommodate that by kicking off reads for synchronous devices for each iteration.
Signed-off-by Christoph Hellwig <[email protected]>--- mm/page_io.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)mm/page_io.c +10
-2@@ -358,7 +358,16 @@ static void swap_add_folio(struct swap_io_ctx *ctx, struct folio *folio, int rw) }bvec_set_folio (&sio->bvecs[sio->nr_bvecs], folio,folio_size (folio),0 ); sio->len +=folio_size (folio); -if (++sio->nr_bvecs ==ARRAY_SIZE (sio->bvecs)) { + +/* +* Write out the iocb if we filled it, or if the device is synchronous. +* +* The latter is to work around expectations in the classic LRU code +* which make synchronous clearing of the folio writeback flag in the +* reclaim path beneficial. +*/ +if (++sio->nr_bvecs ==ARRAY_SIZE (sio->bvecs) || + (rw ==WRITE && (sis->flags &SWP_SYNCHRONOUS_IO ))) {if (rw ==WRITE )swap_write_submit (ctx);else --
Quoted 2 lines
2.53.0
Looks good to me, thank for the update.
PATCH 4/7 mm/swap: remove count_swpout_vm_event
In reply to Christoph HellwigThere is only one caller left, so merge it into that.
--- mm/page_io.c | 24 ++++++++++-------------- 1 file changed, 10 insertions(+), 14 deletions(-)
mm/page_io.c +10 -14
@@ -254,19 +254,6 @@ int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio)return ret; } -static inline void count_swpout_vm_event (struct folio *folio) -{ -#ifdef CONFIG_TRANSPARENT_HUGEPAGE -if (unlikely (folio_test_pmd_mappable (folio))) { -count_memcg_folio_events (folio,THP_SWPOUT ,1 ); -count_vm_event (THP_SWPOUT ); - } -#endif -count_mthp_stat (folio_order (folio),MTHP_STAT_SWPOUT ); -count_memcg_folio_events (folio,PSWPOUT ,folio_nr_pages (folio)); -count_vm_events (PSWPOUT ,folio_nr_pages (folio)); -} -#if defined(CONFIG_MEMCG ) && defined(CONFIG_BLK_CGROUP )static struct cgroup_subsys_state *folio_memcg_blkg_css (struct folio *folio) {
@@ -397,7 +384,16 @@ void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio) {VM_BUG_ON_FOLIO (!folio_test_swapcache (folio), folio); -count_swpout_vm_event (folio); +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +if (unlikely (folio_test_pmd_mappable (folio))) { +count_memcg_folio_events (folio,THP_SWPOUT ,1 ); +count_vm_event (THP_SWPOUT ); + } +#endif +count_mthp_stat (folio_order (folio),MTHP_STAT_SWPOUT ); +count_memcg_folio_events (folio,PSWPOUT ,folio_nr_pages (folio)); +count_vm_events (PSWPOUT ,folio_nr_pages (folio)); +folio_start_writeback (folio);folio_unlock (folio);swap_add_folio (ctx, folio,WRITE );
Signature · 1 line
2.53.0
PATCH 5/7 mm/swap: use swap_ops to register swap device's methods
In reply to Christoph HellwigThis simplifies codes and makes logic clearer. And also makes later any new swap device type being added easier to handle.
Currently there are two types of swap devices: fs and bdev.
--- include/linux/swap.h | 1 + mm/page_io.c | 68 ++++++++++++++++++++++++++++---------------- mm/swap.h | 10 +++++++ mm/swapfile.c | 4 +++ 4 files changed, 58 insertions(+), 25 deletions(-)
include/linux/swap.h +1 -0
@@ -276,6 +276,7 @@ struct swap_info_struct {struct work_struct reclaim_work;/* reclaim worker */ struct list_head discard_clusters;/* discard clusters list */ struct plist_node avail_list;/* entry in swap_avail_head */ +const struct swap_ops *ops; };static inline swp_entry_t page_swap_entry (struct page *page)
mm/page_io.c +43 -25
96 lines not on this page · raw message
mm/swap.h +10 -0
@@ -82,6 +82,13 @@ struct swap_io_ctx {struct swap_info_struct *sis; }; +struct swap_ops { +bool (*can_merge)(struct folio *folio,struct folio *prev_folio, +size_t prev_folio_size,int rw); +void (*submit_write)(struct swap_io_ctx *ctx); +void (*submit_read)(struct swap_io_ctx *ctx); +}; +#ifdef CONFIG_SWAP #include <linux/swapops.h> /* for swp_offset */ #include <linux/blk_types.h> /* for bio_end_io_t */
@@ -469,6 +476,9 @@ static inline unsigned int folio_swap_flags(struct folio *folio)#endif /* CONFIG_SWAP */ +extern const struct swap_ops swap_bdev_ops; +extern const struct swap_ops swap_fs_ops; +int shmem_writeout (struct swap_io_ctx *ctx,struct folio *folio,struct list_head *folio_list);
mm/swapfile.c +4 -0
@@ -2950,6 +2950,8 @@ static int setup_swap_extents(struct swap_info_struct *sis,if (ret)return ret; + sis->ops = &swap_bdev_ops; +if (S_ISBLK (inode->i_mode)) { ret =add_swap_extent (sis,0 , sis->max,0 ); *span = sis->pages;
@@ -2960,6 +2962,8 @@ static int setup_swap_extents(struct swap_info_struct *sis, ret = mapping->a_ops->swap_activate (sis, swap_file, span);if (ret <0 )return ret; +if (sis->flags &SWP_FS_OPS ) + sis->ops = &swap_fs_ops; sis->flags |=SWP_ACTIVATED ;return ret; }
Signature · 1 line
2.53.0
PATCH 6/7 mm/swap: remove SWP_FS_OPS
In reply to Christoph HellwigProvide a swap_fs_activate helper that directly sets up swap_fs_ops, and a flag in struct swap_ops to indicate of NOFS swapping is allowed.
--- Documentation/filesystems/locking.rst | 5 +++-- Documentation/filesystems/vfs.rst | 4 ++-- fs/nfs/file.c | 4 +--- fs/smb/client/file.c | 4 +--- include/linux/swap.h | 6 +++++- mm/page_io.c | 10 +++++++++- mm/swap.h | 22 ++++++++++------------ mm/swapfile.c | 2 -- mm/vmscan.c | 15 +++++++-------- 9 files changed, 38 insertions(+), 34 deletions(-)
Documentation/filesystems/locking.rst +3 -2
@@ -355,13 +355,14 @@ should perform any validation and preparation necessary to ensure that writes can be performed with minimal memory allocation. It should call add_swap_extent(), or the helper iomap_swapfile_activate(), and return the number of extents added. If IO should be submitted through -->swap_rw(), it should set SWP_FS_OPS, otherwise IO will be submitted +->swap_rw(), it should call swap_fs_activate, otherwise IO will be submitted directly to the block device ``sis->bdev``. ->swap_deactivate() will be called in the sys_swapoff() path after ->swap_activate() returned success. -->swap_rw will be called for swap IO if SWP_FS_OPS was set by ->swap_activate(). +->swap_rw will be called for swap IO if swap_fs_activate was called by +->swap_activate(). file_lock_operations ====================
Documentation/filesystems/vfs.rst +2 -2
@@ -977,7 +977,7 @@ cache in your filesystem. The following members are defined: can be performed with minimal memory allocation. It should call add_swap_extent(), or the helper iomap_swapfile_activate(), and return the number of extents added. If IO should be submitted - through ->swap_rw(), it should set SWP_FS_OPS, otherwise IO will + through ->swap_rw(), it should call swap_fs_activate, otherwise IO will be submitted directly to the block device ``sis->bdev``. ``swap_deactivate``
@@ -985,7 +985,7 @@ cache in your filesystem. The following members are defined: successful. ``swap_rw`` - Called to read or write swap pages when SWP_FS_OPS is set. + Called to read or write swap pages when swap_fs_activate was called. The File Object ===============
fs/nfs/file.c +1 -3
@@ -597,7 +597,7 @@ static int nfs_swap_activate(struct swap_info_struct *sis, struct file *file, ret =rpc_clnt_swap_activate (clnt);if (ret)return ret; - ret =add_swap_extent (sis,0 , sis->max,0 ); + ret =swap_fs_activate (sis);if (ret <0 ) {rpc_clnt_swap_deactivate (clnt);return ret;
@@ -607,8 +607,6 @@ static int nfs_swap_activate(struct swap_info_struct *sis, struct file *file,if (cl->rpc_ops->enable_swap) cl->rpc_ops->enable_swap (inode); - - sis->flags |=SWP_FS_OPS ;return ret; }
fs/smb/client/file.c +1 -3
@@ -3387,9 +3387,7 @@ static int cifs_swap_activate(struct swap_info_struct *sis,* but we could add call to grab a byte range lock to prevent others * from reading or writing the file */ - - sis->flags |=SWP_FS_OPS ; -return add_swap_extent (sis,0 , sis->max,0 ); +return swap_fs_activate (sis); }static void cifs_swap_deactivate (struct file *file)
include/linux/swap.h +5 -1
@@ -202,7 +202,6 @@ enum {SWP_SOLIDSTATE = (1 <<4 ),/* blkdev seeks are cheap */ SWP_BLKDEV = (1 <<6 ),/* its a block device */ SWP_ACTIVATED = (1 <<7 ),/* set after swap_activate success */ -SWP_FS_OPS = (1 <<8 ),/* swapfile operations go through fs */ SWP_AREA_DISCARD = (1 <<9 ),/* single-time swap area discards */ SWP_PAGE_DISCARD = (1 <<10 ),/* freed swap page-cluster discards */ SWP_STABLE_WRITES = (1 << 11), /* no overwrite PG_writeback pages */
@@ -336,6 +335,7 @@ extern void __meminit kswapd_stop(int nid);#ifdef CONFIG_SWAP +int swap_fs_activate (struct swap_info_struct *sis);int add_swap_extent (struct swap_info_struct *sis,unsigned long start_page,unsigned long nr_pages,sector_t start_block); int generic_swapfile_activate(struct swap_info_struct *, struct file *,
@@ -461,6 +461,10 @@ static inline bool folio_free_swap(struct folio *folio)return false; } +static inline int swap_fs_activate (struct swap_info_struct *sis) +{ +return -EINVAL ; +}static inline int add_swap_extent (struct swap_info_struct *sis,unsigned long start_page,unsigned long nr_pages,sector_t start_block)
mm/page_io.c +9 -1
@@ -686,12 +686,20 @@ static bool swap_fs_can_merge(struct folio *folio, struct folio *prev_folio,swap_dev_pos (prev_folio->swap) + prev_folio_size; } -const struct swap_ops swap_fs_ops = { +static const struct swap_ops swap_fs_ops = { + .flags =SWAP_OPS_F_REQUIRE_NOFS , .submit_write = swap_fs_submit_write, .submit_read = swap_fs_submit_read, .can_merge = swap_fs_can_merge, }; +int swap_fs_activate (struct swap_info_struct *sis) +{ + sis->ops = &swap_fs_ops; +return add_swap_extent (sis,0 , sis->max,0 ); +} +EXPORT_SYMBOL_GPL (swap_fs_activate); +void swap_write_submit (struct swap_io_ctx *ctx) {if (!ctx->sio)
mm/swap.h +10 -12
@@ -82,7 +82,17 @@ struct swap_io_ctx {struct swap_info_struct *sis; }; +/* +* SWAP_OPS_F_REQUIRE_NOFS: +* When set, all reclaim operations must operated as GFS_NOFS and not +* just GFP_NOIO, as GFP_NOIO allocations could recourse into the +* file system backing this swap file. +*/ +#define SWAP_OPS_F_REQUIRE_NOFS (1U << 0) +struct swap_ops { +unsigned int flags; +bool (*can_merge)(struct folio *folio,struct folio *prev_folio,size_t prev_folio_size,int rw);void (*submit_write)(struct swap_io_ctx *ctx);
@@ -333,11 +343,6 @@ struct folio *swapin_sync(swp_entry_t entry, gfp_t flag, unsigned long orders,void swap_update_readahead (struct folio *folio,struct vm_area_struct *vma,unsigned long addr); -static inline unsigned int folio_swap_flags (struct folio *folio) -{ -return __swap_entry_to_info (folio->swap)->flags; -} -#else /* CONFIG_SWAP */ static inline struct swap_cluster_info *swap_cluster_lock (struct swap_info_struct *si,pgoff_t offset,bool irq)
@@ -468,16 +473,9 @@ static inline void __swap_cache_replace_folio(struct swap_cluster_info *ci, struct folio *old, struct folio *new) { } - -static inline unsigned int folio_swap_flags (struct folio *folio) -{ -return 0 ; -} -#endif /* CONFIG_SWAP */ extern const struct swap_ops swap_bdev_ops; -extern const struct swap_ops swap_fs_ops;int shmem_writeout (struct swap_io_ctx *ctx,struct folio *folio,struct list_head *folio_list);
mm/swapfile.c +0 -2
@@ -2962,8 +2962,6 @@ static int setup_swap_extents(struct swap_info_struct *sis, ret = mapping->a_ops->swap_activate (sis, swap_file, span);if (ret <0 )return ret; -if (sis->flags &SWP_FS_OPS ) - sis->ops = &swap_fs_ops; sis->flags |=SWP_ACTIVATED ;return ret; }
mm/vmscan.c +7 -8
@@ -1033,16 +1033,15 @@ static bool may_enter_fs(struct folio *folio, gfp_t gfp_mask) {if (gfp_mask & __GFP_FS)return true; -if (!folio_test_swapcache (folio) || !(gfp_mask & __GFP_IO)) -return false;/* -* We can "enter_fs" for swap-cache with only __GFP_IO -* providing this isn't SWP_FS_OPS. -* ->flags can be updated non-atomically, -* but that will never affect SWP_FS_OPS, so the data_race -* is safe. +* We can "enter_fs" for swap-cache with only __GFP_IO unless backed by +* a swapfile that requires GFP_NOFS I/O. */ -return !data_race (folio_swap_flags (folio) &SWP_FS_OPS ); +if (folio_test_swapcache (folio) && (gfp_mask & __GFP_IO) && + !(__swap_entry_to_info (folio->swap)->ops->flags & +SWAP_OPS_F_REQUIRE_NOFS )) +return true; +return false; }/*
Signature · 1 line
2.53.0
PATCH 7/7 mm/vmstat: add NRSWP{IN,OUT} counters
In reply to Christoph HellwigCount how many swap I/Os we cause. Due to batching this can be different than the current counter number of pages written/read, and tracking this information is useful to see how efficient the batching is.
The counters are added at the end of enum vm_event_item and the vmstat_text array under the assumption that the order of fields in /proc/vmstat is an ABI. If that is not the case, they could be grouped with the other swap counters.
--- include/linux/vm_event_item.h | 4 ++++ mm/page_io.c | 2 ++ mm/vmstat.c | 6 +++++- 3 files changed, 11 insertions(+), 1 deletion(-)
include/linux/vm_event_item.h +4 -0
@@ -175,6 +175,10 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, KSTACK_REST,#endif #endif /* CONFIG_DEBUG_STACK_USAGE */ +#ifdef CONFIG_SWAP +NRSWPIN , +NRSWPOUT , +#endif /* CONFIG_SWAP */NR_VM_EVENT_ITEMS };
mm/page_io.c +2 -0
@@ -704,6 +704,7 @@ void swap_write_submit(struct swap_io_ctx *ctx) {if (!ctx->sio)return ; +count_vm_events (NRSWPOUT ,1 ); ctx->sis->ops->submit_write (ctx); ctx->sio =NULL ; ctx->sis =NULL ;
@@ -713,6 +714,7 @@ void swap_read_submit(struct swap_io_ctx *ctx) {if (!ctx->sio)return ; +count_vm_events (NRSWPIN ,1 ); ctx->sis->ops->submit_read (ctx); ctx->sio =NULL ; ctx->sis =NULL ;
mm/vmstat.c +5 -1
@@ -1489,7 +1489,11 @@ const char * const vmstat_text[] = {#if THREAD_SIZE >65536 [I(KSTACK_REST)] = "kstack_rest",#endif -#endif +#endif /* CONFIG_DEBUG_STACK_USAGE */ +#ifdef CONFIG_SWAP + [I(NRSWPIN)] = "nrswpin", + [I(NRSWPOUT)] = "nrswpout", +#endif /* CONFIG_SWAP */ #undef I#endif /* CONFIG_VM_EVENT_COUNTERS */ };
Signature · 1 line
2.53.0
Andrew Morton
maintainerQuoted 8 lines · On Mon, 13 Jul 2026 11:33:37 +0200 Christoph Hellwig <[email protected]> wrote:
this series makes use of the
swap_iocbfor block as well so that it doesn't do inefficient single-bio I/O, and then rebases theswap_opsfrom Baoquan on top of the now very different method structure.When running doing kernels builds, which is a workload that doesn't really do much THP anonymous memory it still gets 2x clustering for writeout and 1.2x for reading back swap in. The overall times do not actually change, though.
Thanks. Sashiko review might have found a couple of deadly issues in
drivers/gpu/drm/i915/gem/i915_gem_shmem.c drivers/gpu/drm/ttm/ttm_backup.c
so let me add a zillions cc's.
https://sashiko.dev/#/patchset/[email protected]
Quoted 11 lines · Andrew Morton
Quoted 8 lines · On Mon, 13 Jul 2026 11:33:37 +0200 Christoph Hellwig <[email protected]> wrote:
this series makes use of the
swap_iocbfor block as well so that it doesn't do inefficient single-bio I/O, and then rebases theswap_opsfrom Baoquan on top of the now very different method structure.When running doing kernels builds, which is a workload that doesn't really do much THP anonymous memory it still gets 2x clustering for writeout and 1.2x for reading back swap in. The overall times do not actually change, though.
Quoted 6 lines
Thanks. Sashiko review might have found a couple of deadly issues in
drivers/gpu/drm/i915/gem/i915_gem_shmem.c drivers/gpu/drm/ttm/ttm_backup.cso let me add a zillions cc's.
Well, I mentioned this before, as did you. But it is all existing issues. The proper fix would of course be to not give users such low-level access to shmem writeback..
On the other Sashiko comment: the bi_vcnt doesn't make any sense, bi_vcnt is only for the submitters convenience.