Skip to contentSkip to conversation

mm/shmem: remove truncated folios from the page cache in batches

PATCH Applied Qiliang Yuan 3 participants 3 messages Nov 12 – Nov 13

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 mm/shmem: remove truncated folios from the page cache in batches

truncate_inode_pages_range() takes the folios it finds out of the page cache with delete_from_page_cache_batch(), which takes the inode's i_lock and the i_pages lock once for each batch of up to FOLIO_BATCH_SIZE folios.

shmem_undo_range() calls truncate_inode_folio() on each folio instead, so filemap_remove_folio() takes and drops both locks once per folio. Truncating, punching or freeing a tmpfs file or memfd made of small folios, such as a large /dev/shm segment, pays for that on every page.

Clean each folio up as truncate_inode_folio() does and, once the swap entries are gone, pass the batch to delete_from_page_cache_batch(), which needs truncate_cleanup_folio() outside mm/truncate.c. Undoing a failed fallocate() only removes the folios that are not uptodate, so keep removing those one at a time.

Dropping an 8 GiB file of 4K folios in a 32 vCPU VM takes (median of five runs, averaged over two boots of each kernel):

operation before (ms) after (ms) change

  ftruncate() to 0           1144         901    -21%
  hole punch                 1148         906    -21%
  close() of a memfd         1139         912    -20%
  unlink()                   1142         914    -20%

Files of PMD-sized folios take 18-31 ms with or without this change.

---
V2 -> V3:
- Delete the folios from fbatch directly instead of collecting them in a second folio_batch, which pushed the shmem_undo_range() stack frame past 1024 bytes on mips with clang (reported by the kernel test robot)
- Rebase onto bf87d3fe8b35 in core.git shmem-7.5.misc and update the numbers from a new run
- Drop the Reviewed-by tags from Jan and Baolin, since the loop changed

V1 -> V2:
- Unlock the folios after folio_batch_remove_exceptionals(), as Jan suggested
- Rebase onto v7.4-rc1-132-g05441d697f7f and update the numbers from a new run

v2: https://lore.kernel.org/r/[email protected]
v1: https://lore.kernel.org/r/[email protected]

To see it, fill and remove a large /dev/shm file:

fallocate -l 8G /dev/shm/f && time rm /dev/shm/f

---
 mm/internal.h | 1 +
 mm/shmem.c    | 9 +++++++--
 mm/truncate.c | 2 +-
 3 files changed, 9 insertions(+), 3 deletions(-)
mm/internal.h +1 -0
index 566cbd2dce485..abe64e49a4170 100644
@@ -628,6 +628,7 @@ unsigned find_lock_entries(struct address_space *mapping, pgoff_t *start,
 		pgoff_t end, struct folio_batch *fbatch, pgoff_t *indices);
 unsigned find_get_entries(struct address_space *mapping, pgoff_t *start,
 		pgoff_t end, struct folio_batch *fbatch, pgoff_t *indices);
+void truncate_cleanup_folio(struct folio *folio);
 int truncate_inode_folio(struct address_space *mapping, struct folio *folio);
 bool truncate_inode_partial_folio(struct folio *folio, loff_t lstart,
 		loff_t lend, pgoff_t *pstart, pgoff_t *pend);
mm/shmem.c +7 -2
index ce92e74baeb75..5bcbdd6078a73 100644
@@ -1363,11 +1363,16 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, uoff_t lend,
 				continue;
 			}
 
-			if (!unfalloc || !folio_test_uptodate(folio))
+			if (!unfalloc)
+				truncate_cleanup_folio(folio);
+			else if (!folio_test_uptodate(folio))
 				truncate_inode_folio(mapping, folio);
-			folio_unlock(folio);
 		}
 		folio_batch_remove_exceptionals(&fbatch);
+		if (!unfalloc)
+			delete_from_page_cache_batch(mapping, &fbatch);
+		for (i = 0; i < folio_batch_count(&fbatch); i++)
+			folio_unlock(fbatch.folios[i]);
 		folio_batch_release(&fbatch);
 		cond_resched_tasks_rcu_qs();
 	}
mm/truncate.c +1 -1
index 733d2713a875a..c6bf4aa0b91e8 100644
@@ -151,7 +151,7 @@ EXPORT_SYMBOL_GPL(folio_invalidate);
  * its lock, b) when a concurrent invalidate_mapping_pages got there first and
  * c) when tmpfs swizzles a page between a tmpfs inode and swapper_space.
  */
-static void truncate_cleanup_folio(struct folio *folio)
+void truncate_cleanup_folio(struct folio *folio)
 {
 	if (folio_mapped(folio))
 		unmap_mapping_folio(folio);

---
base-commit: bf87d3fe8b35087c2c043642e76e9f100d61328b
change-id: 20261029-bug-mm-shmem-undo-batch-de001d9c6f1e

Best regards,

Signature · 1 line
Qiliang Yuan <[email protected]>
reviewer Reviewed-by on v3 1/1
#1 lore raw
Quoted 25 lines · Qiliang Yuan

truncate_inode_pages_range() takes the folios it finds out of the page cache with delete_from_page_cache_batch(), which takes the inode's i_lock and the i_pages lock once for each batch of up to FOLIO_BATCH_SIZE folios.

shmem_undo_range() calls truncate_inode_folio() on each folio instead, so filemap_remove_folio() takes and drops both locks once per folio. Truncating, punching or freeing a tmpfs file or memfd made of small folios, such as a large /dev/shm segment, pays for that on every page.

Clean each folio up as truncate_inode_folio() does and, once the swap entries are gone, pass the batch to delete_from_page_cache_batch(), which needs truncate_cleanup_folio() outside mm/truncate.c. Undoing a failed fallocate() only removes the folios that are not uptodate, so keep removing those one at a time.

Dropping an 8 GiB file of 4K folios in a 32 vCPU VM takes (median of five runs, averaged over two boots of each kernel):

   operation           before (ms)  after (ms)  change
   ftruncate() to 0           1144         901    -21%
   hole punch                 1148         906    -21%
   close() of a memfd         1139         912    -20%
   unlink()                   1142         914    -20%
Quoted 4 lines

Files of PMD-sized folios take 18-31 ms with or without this change.

---

LGTM.

maintainer Applied
#2 lore raw
In reply to Qiliang Yuan
Quoted 11 lines · Qiliang Yuan

truncate_inode_pages_range() takes the folios it finds out of the page cache with delete_from_page_cache_batch(), which takes the inode's i_lock and the i_pages lock once for each batch of up to FOLIO_BATCH_SIZE folios.

shmem_undo_range() calls truncate_inode_folio() on each folio instead, so filemap_remove_folio() takes and drops both locks once per folio. Truncating, punching or freeing a tmpfs file or memfd made of small folios, such as a large /dev/shm segment, pays for that on every page.

[...]

Applied, thanks!

tree: mm/core.git
branch: shmem-7.5.misc

Applied patches will appear soon in mm-next, followed by linux-next.

Please report any outstanding bugs that were missed during review, so we can either drop the patch series or decide how to fix it up.

It is encouraged to provide Acked-bys, Reviewed-bys, and Tested-bys even though the patches have already been applied. If possible, patch trailers will be updated.

Note that commit hashes shown below are subject to change due to rebase, trailer updates or similar. Further, this is not a guarantee that the patches will go upstream. If in doubt, please check the listed branch.

[1/1] mm/shmem: remove truncated folios from the page cache in batches

      https://git.kernel.org/mm/core/c/2e0dafbf
Signature · 3 lines
Cheers,

David