mm/shmem: remove truncated folios from the page cache in batches
Subsystems (5)
PATCH v3 1/1 mm/shmem: remove truncated folios from the page cache in batches
truncate_inode_pages_range() takes the folios it finds out of the page cache with delete_from_page_cache_batch(), which takes the inode's i_lock and the i_pages lock once for each batch of up to FOLIO_BATCH_SIZE folios.
shmem_undo_range() calls truncate_inode_folio() on each folio instead, so filemap_remove_folio() takes and drops both locks once per folio. Truncating, punching or freeing a tmpfs file or memfd made of small folios, such as a large /dev/shm segment, pays for that on every page.
Clean each folio up as truncate_inode_folio() does and, once the swap entries are gone, pass the batch to delete_from_page_cache_batch(), which needs truncate_cleanup_folio() outside mm/truncate.c. Undoing a failed fallocate() only removes the folios that are not uptodate, so keep removing those one at a time.
Dropping an 8 GiB file of 4K folios in a 32 vCPU VM takes (median of five runs, averaged over two boots of each kernel):
operation before (ms) after (ms) change
ftruncate() to 0 1144 901 -21% hole punch 1148 906 -21% close() of a memfd 1139 912 -20% unlink() 1142 914 -20%
Files of PMD-sized folios take 18-31 ms with or without this change.
---
V2 -> V3:
- Delete the folios from fbatch directly instead of collecting them in a second folio_batch, which pushed the shmem_undo_range() stack frame past 1024 bytes on mips with clang (reported by the kernel test robot)
- Rebase onto bf87d3fe8b35 in core.git shmem-7.5.misc and update the numbers from a new run
- Drop the Reviewed-by tags from Jan and Baolin, since the loop changed
V1 -> V2:
- Unlock the folios after folio_batch_remove_exceptionals(), as Jan suggested
- Rebase onto v7.4-rc1-132-g05441d697f7f and update the numbers from a new run
v2: https://lore.kernel.org/r/[email protected]
v1: https://lore.kernel.org/r/[email protected]
To see it, fill and remove a large /dev/shm file:
fallocate -l 8G /dev/shm/f && time rm /dev/shm/f
--- mm/internal.h | 1 + mm/shmem.c | 9 +++++++-- mm/truncate.c | 2 +- 3 files changed, 9 insertions(+), 3 deletions(-)
mm/internal.h +1 -0
@@ -628,6 +628,7 @@ unsigned find_lock_entries(struct address_space *mapping, pgoff_t *start, pgoff_t end, struct folio_batch *fbatch, pgoff_t *indices);unsigned find_get_entries (struct address_space *mapping,pgoff_t *start,pgoff_t end,struct folio_batch *fbatch,pgoff_t *indices); +void truncate_cleanup_folio (struct folio *folio);int truncate_inode_folio (struct address_space *mapping,struct folio *folio);bool truncate_inode_partial_folio (struct folio *folio,loff_t lstart,loff_t lend,pgoff_t *pstart,pgoff_t *pend);
mm/shmem.c +7 -2
@@ -1363,11 +1363,16 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, uoff_t lend,continue ; } -if (!unfalloc || !folio_test_uptodate (folio)) +if (!unfalloc) +truncate_cleanup_folio (folio); +else if (!folio_test_uptodate (folio))truncate_inode_folio (mapping, folio); -folio_unlock (folio); }folio_batch_remove_exceptionals (&fbatch); +if (!unfalloc) +delete_from_page_cache_batch (mapping, &fbatch); +for (i =0 ; i <folio_batch_count (&fbatch); i++) +folio_unlock (fbatch.folios[i]);folio_batch_release (&fbatch);cond_resched_tasks_rcu_qs (); }
mm/truncate.c +1 -1
@@ -151,7 +151,7 @@ EXPORT_SYMBOL_GPL(folio_invalidate);* its lock, b) when a concurrent invalidate_mapping_pages got there first and * c) when tmpfs swizzles a page between a tmpfs inode and swapper_space. */ -static void truncate_cleanup_folio (struct folio *folio) +void truncate_cleanup_folio (struct folio *folio) {if (folio_mapped (folio))unmap_mapping_folio (folio);
---
base-commit: bf87d3fe8b35087c2c043642e76e9f100d61328b
change-id: 20261029-bug-mm-shmem-undo-batch-de001d9c6f1e
Best regards,
Signature · 1 line
Qiliang Yuan <[email protected]>
Quoted 25 lines · Qiliang Yuan
truncate_inode_pages_range()takes the folios it finds out of the page cache withdelete_from_page_cache_batch(), which takes the inode'si_lockand thei_pageslock once for each batch of up toFOLIO_BATCH_SIZEfolios.
shmem_undo_range()callstruncate_inode_folio()on each folio instead, sofilemap_remove_folio()takes and drops both locks once per folio. Truncating, punching or freeing a tmpfs file or memfd made of small folios, such as a large/dev/shmsegment, pays for that on every page.Clean each folio up as
truncate_inode_folio()does and, once the swap entries are gone, pass the batch todelete_from_page_cache_batch(), which needstruncate_cleanup_folio()outsidemm/truncate.c. Undoing a failedfallocate()only removes the folios that are not uptodate, so keep removing those one at a time.Dropping an 8 GiB file of 4K folios in a 32 vCPU VM takes (median of five runs, averaged over two boots of each kernel):
operation before (ms) after (ms) change ftruncate() to 0 1144 901 -21% hole punch 1148 906 -21% close() of a memfd 1139 912 -20% unlink() 1142 914 -20%
Quoted 4 lines
Files of PMD-sized folios take 18-31 ms with or without this change.
Signed-off-by Qiliang Yuan <[email protected]>---
LGTM.
Quoted 11 lines · Qiliang Yuan
truncate_inode_pages_range()takes the folios it finds out of the page cache withdelete_from_page_cache_batch(), which takes the inode'si_lockand thei_pageslock once for each batch of up toFOLIO_BATCH_SIZEfolios.
shmem_undo_range()callstruncate_inode_folio()on each folio instead, sofilemap_remove_folio()takes and drops both locks once per folio. Truncating, punching or freeing a tmpfs file or memfd made of small folios, such as a large/dev/shmsegment, pays for that on every page.[...]
Applied, thanks!
tree: mm/core.git
branch: shmem-7.5.misc
Applied patches will appear soon in mm-next, followed by linux-next.
Please report any outstanding bugs that were missed during review, so we can either drop the patch series or decide how to fix it up.
It is encouraged to provide Acked-bys, Reviewed-bys, and Tested-bys even though the patches have already been applied. If possible, patch trailers will be updated.
Note that commit hashes shown below are subject to change due to rebase, trailer updates or similar. Further, this is not a guarantee that the patches will go upstream. If in doubt, please check the listed branch.
[1/1] mm/shmem: remove truncated folios from the page cache in batches
https://git.kernel.org/mm/core/c/2e0dafbf
Signature · 3 lines
Cheers, David