guest_memfd: Track amount of memory allocated on inode
Subsystems (8)
cover guest_memfd: Track amount of memory allocated on inode
Hi,
Currently, guest_memfd doesn't update inode's i_blocks or i_bytes at all. Hence, st_blocks in the struct populated by a userspace fstat() call on a guest_memfd will always be 0. This patch series makes guest_memfd track the amount of memory allocated on an inode, which allows fstat() to accurately report that on requests from userspace.
The inode's i_blocks and i_bytes fields are updated when the folio is associated or disassociated from the guest_memfd inode, which are at allocation and truncation times respectively.
To update inode fields at truncation time, this series implements a custom truncation function for guest_memfd. An alternative would be to update truncate_inode_pages_range() to return the number of bytes truncated or add/use some hook.
Implementing a custom truncation function was chosen to provide flexibility for handling truncations in future when guest_memfd supports sources of pages other than the buddy allocator. This approach of a custom truncation function also aligns with shmem, which has a custom shmem_truncate_range().
To update inode fields at allocation time, kvm_gmem_get_folio() is simply augmented such that when a folio is added to the filemap, the size of the folio is updated into inode fields.
The second patch, to use filemap_alloc_folio() during allocation of guest_memfd folios, was written as a debugging step to resolve a bug found by syzbot [1], but turned out to not be the fix. I include it here because it cleans up the allocation process and provides a nice foundation for updating inode fields during allocations.
The first patch was separately submitted [2], and provided here since it is a prerequisite simplication before application of the second patch.
[1] https://lore.kernel.org/all/29c347bde68ec027259654e8e85371307edf7058.1770148108.git.ackerleytng@google.com/
[2] https://lore.kernel.org/all/[email protected]/
Ackerley Tng (10):
KVM: guest_memfd: Don't set FGP_ACCESSED when getting folios
KVM: guest_memfd: Directly allocate folios with filemap_alloc_folio()
mm: truncate: Expose preparation steps for
truncate_inode_pages_final()
KVM: guest_memfd: Implement evict_inode for guest_memfd
mm: Export unmap_mapping_folio() for KVM
mm: filemap: Export filemap_remove_folio()
KVM: guest_memfd: Implement custom truncation function
KVM: guest_memfd: Track amount of memory allocated on inode
KVM: selftests: Wrap fstat() to assert success
KVM: selftests: Test that st_blocks is updated on allocation
include/linux/mm.h | 3 + mm/filemap.c | 2 + mm/internal.h | 2 - mm/memory.c | 2 + mm/truncate.c | 21 +++- .../testing/selftests/kvm/guest_memfd_test.c | 32 +++-- .../selftests/kvm/include/kvm_syscalls.h | 2 + virt/kvm/guest_memfd.c | 116 +++++++++++++++--- 8 files changed, 149 insertions(+), 31 deletions(-)
base-commit: b1195183ed42f1522fae3fe44ebee3af437aa000 --
2.53.0.345.g96ddfc5eaa-goog
RFC PATCH 1/10 KVM: guest_memfd: Don't set FGP_ACCESSED when getting folios
guest_memfd folios don't care about accessed flags since the memory is unevictable and there is no storage to write back to, hence, cleanup the allocation path by not setting FGP_ACCESSED.
--- virt/kvm/guest_memfd.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
virt/kvm/guest_memfd.c +2 -3
@@ -126,14 +126,13 @@ static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index)* Fast-path: See if folio is already present in mapping to avoid * policy_lookup. */ - folio =__filemap_get_folio (inode->i_mapping, index, -FGP_LOCK |FGP_ACCESSED ,0 ); + folio =filemap_lock_folio (inode->i_mapping, index);if (!IS_ERR (folio))return folio; policy =mpol_shared_policy_lookup (&GMEM_I (inode)->policy, index); folio =__filemap_get_folio_mpol (inode->i_mapping, index, -FGP_LOCK |FGP_ACCESSED |FGP_CREAT , +FGP_LOCK |FGP_CREAT ,mapping_gfp_mask (inode->i_mapping), policy);mpol_cond_put (policy);
Signature · 1 line
2.53.0.345.g96ddfc5eaa-goog
RFC PATCH 2/10 KVM: guest_memfd: Directly allocate folios with filemap_alloc_folio()
In reply to Ackerley Tng__filemap_get_folio_mpol() is parametrized by a bunch of GFP flags, which adds complexity for the reader. Since guest_memfd doesn't meaningfully use any of the other FGP flags, undo that complexity by directly calling filemap_alloc_folio().
Directly calling filemap_alloc_folio() also allows the order of 0 to be explicitly specified, which is the only order guest_memfd supports. This is easier to understand, and removes the chance of anything else being able to unintentionally influence allocated folio size.
--- virt/kvm/guest_memfd.c | 51 +++++++++++++++++++++++++++++------------- 1 file changed, 36 insertions(+), 15 deletions(-)
virt/kvm/guest_memfd.c +36 -15
@@ -107,6 +107,39 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,return __kvm_gmem_prepare_folio (kvm, slot, index, folio); } +static struct folio *__kvm_gmem_get_folio (struct inode *inode,pgoff_t index) +{ +/* TODO: Support huge pages. */ +struct mempolicy *policy; +struct folio *folio; +gfp_t gfp; +int ret; + +/* +* Fast-path: See if folio is already present in mapping to avoid +* policy_lookup. +*/ + folio =filemap_lock_folio (inode->i_mapping, index); +if (!IS_ERR (folio)) +return folio; + + gfp =mapping_gfp_mask (inode->i_mapping); + + policy =mpol_shared_policy_lookup (&GMEM_I (inode)->policy, index); + folio =filemap_alloc_folio (gfp,0 , policy); +mpol_cond_put (policy); +if (!folio) +return ERR_PTR (-ENOMEM ); + + ret =filemap_add_folio (inode->i_mapping, folio, index, gfp); +if (ret) { +folio_put (folio); +return ERR_PTR (ret); + } + +return folio; +} +/* * Returns a locked folio on success. The caller is responsible for * setting the up-to-date flag before the memory is mapped into the guest.
@@ -118,23 +151,11 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,*/ static struct folio *kvm_gmem_get_folio (struct inode *inode,pgoff_t index) { -/* TODO: Support huge pages. */ -struct mempolicy *policy;struct folio *folio; -/* -* Fast-path: See if folio is already present in mapping to avoid -* policy_lookup. -*/ - folio =filemap_lock_folio (inode->i_mapping, index); -if (!IS_ERR (folio)) -return folio; - - policy =mpol_shared_policy_lookup (&GMEM_I (inode)->policy, index); - folio =__filemap_get_folio_mpol (inode->i_mapping, index, -FGP_LOCK |FGP_CREAT , -mapping_gfp_mask (inode->i_mapping), policy); -mpol_cond_put (policy); +do { + folio =__kvm_gmem_get_folio (inode, index); + }while (PTR_ERR (folio) == -EEXIST );/* * External interfaces like kvm_gmem_get_pfn() support dealing
Signature · 1 line
2.53.0.345.g96ddfc5eaa-goog
RFC PATCH 3/10 mm: truncate: Expose preparation steps for truncate_inode_pages_final()
In reply to Ackerley TngExpose preparation steps for truncate_inode_pages_final() to allow preparation steps to be shared by filesystems that want to implement truncation differently.
This preparation function will be used by guest_memfd in a later patch.
--- include/linux/mm.h | 1 + mm/truncate.c | 21 +++++++++++++++++++-- 2 files changed, 20 insertions(+), 2 deletions(-)
include/linux/mm.h +1 -0
@@ -3732,6 +3732,7 @@ extern unsigned long vm_unmapped_area(struct vm_unmapped_area_info *info);void truncate_inode_pages (struct address_space *mapping,loff_t lstart);void truncate_inode_pages_range (struct address_space *mapping,loff_t lstart,uoff_t lend); +void truncate_inode_pages_final_prepare (struct address_space *mapping);void truncate_inode_pages_final (struct address_space *mapping);/* generic vm_area_ops exported for stackable file systems */
mm/truncate.c +19 -2
@@ -487,7 +487,9 @@ void truncate_inode_pages(struct address_space *mapping, loff_t lstart)EXPORT_SYMBOL (truncate_inode_pages);/** -* truncate_inode_pages_final - truncate *all* pages before inode dies +* truncate_inode_pages_final_prepare - Prepare the mapping for final +* truncation but not actually truncate the inode pages. This could be +* used by filesystems which want to add custom truncation of folios. * @mapping: mapping to truncate * * Called under (and serialized by) inode->i_rwsem.
@@ -495,7 +497,7 @@ EXPORT_SYMBOL(truncate_inode_pages);* Filesystems have to use this in the .evict_inode path to inform the * VM that this is the final truncate and the inode is going away. */ -void truncate_inode_pages_final (struct address_space *mapping) +void truncate_inode_pages_final_prepare (struct address_space *mapping) {/* * Page reclaim can not participate in regular inode lifetime
@@ -516,6 +518,21 @@ void truncate_inode_pages_final(struct address_space *mapping)xa_lock_irq (&mapping->i_pages);xa_unlock_irq (&mapping->i_pages); } +} +EXPORT_SYMBOL (truncate_inode_pages_final_prepare); + +/** +* truncate_inode_pages_final - truncate *all* pages before inode dies +* @mapping: mapping to truncate +* +* Called under (and serialized by) inode->i_rwsem. +* +* Filesystems have to use this in the .evict_inode path to inform the +* VM that this is the final truncate and the inode is going away. +*/ +void truncate_inode_pages_final (struct address_space *mapping) +{ +truncate_inode_pages_final_prepare (mapping);truncate_inode_pages (mapping,0 ); }
Signature · 1 line
2.53.0.345.g96ddfc5eaa-goog
RFC PATCH 4/10 KVM: guest_memfd: Implement evict_inode for guest_memfd
In reply to Ackerley TngIn a later patch, guest_memfd will be using a custom truncation routine. This is a preparatory patch, which implements .evict_inode for guest_memfd, but just performs exactly what would have been done if .evict_inode were not implemented.
No functional change intended.
--- virt/kvm/guest_memfd.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
virt/kvm/guest_memfd.c +12 -0
@@ -978,11 +978,23 @@ static void kvm_gmem_free_inode(struct inode *inode)kmem_cache_free (kvm_gmem_inode_cachep,GMEM_I (inode)); } +static void kvm_gmem_evict_inode (struct inode *inode) +{ +struct address_space *mapping = inode->i_mapping; + +truncate_inode_pages_final_prepare (mapping); + +truncate_inode_pages_range (mapping,0 , inode->i_size); + +clear_inode (inode); +} +static const struct super_operations kvm_gmem_super_operations = { .statfs = simple_statfs, .alloc_inode = kvm_gmem_alloc_inode, .destroy_inode = kvm_gmem_destroy_inode, .free_inode = kvm_gmem_free_inode, + .evict_inode = kvm_gmem_evict_inode, };static int kvm_gmem_init_fs_context (struct fs_context *fc)
Signature · 1 line
2.53.0.345.g96ddfc5eaa-goog
RFC PATCH 5/10 mm: Export unmap_mapping_folio() for KVM
In reply to Ackerley Tngguest_memfd needs a way to unmap a folio from all userspace processes. This is required as part of a folio's truncation process. The function unmap_mapping_folio() provides exactly this functionality.
Move its declaration from the internal mm/internal.h to the public include/linux/mm.h and export the symbol.
unmap_mapping_folio() will be used by guest_memfd in a later patch to implement a custom truncation function.
No functional change intended.
--- include/linux/mm.h | 2 ++ mm/internal.h | 2 -- mm/memory.c | 2 ++ 3 files changed, 4 insertions(+), 2 deletions(-)
include/linux/mm.h +2 -0
@@ -2690,6 +2690,7 @@ extern vm_fault_t handle_mm_fault(struct vm_area_struct *vma,extern int fixup_user_fault (struct mm_struct *mm,unsigned long address,unsigned int fault_flags,bool *unlocked); +void unmap_mapping_folio (struct folio *folio);void unmap_mapping_pages (struct address_space *mapping,pgoff_t start,pgoff_t nr,bool even_cows); void unmap_mapping_range(struct address_space *mapping,
@@ -2710,6 +2711,7 @@ static inline int fixup_user_fault(struct mm_struct *mm, unsigned long address,BUG ();return -EFAULT ; } +static inline void unmap_mapping_folio (struct folio *folio) { }static inline void unmap_mapping_pages (struct address_space *mapping,pgoff_t start,pgoff_t nr,bool even_cows) { } static inline void unmap_mapping_range(struct address_space *mapping,
mm/internal.h +0 -2
@@ -953,7 +953,6 @@ static inline bool free_area_empty(struct free_area *area, int migratetype)struct anon_vma *folio_anon_vma (const struct folio *folio);#ifdef CONFIG_MMU -void unmap_mapping_folio (struct folio *folio);extern long populate_vma_page_range (struct vm_area_struct *vma,unsigned long start,unsigned long end,int *locked); extern long faultin_page_range(struct mm_struct *mm, unsigned long start,
@@ -1131,7 +1130,6 @@ static inline struct file *maybe_unlock_mmap_for_io(struct vm_fault *vmf, return fpin; }#else /* !CONFIG_MMU */ -static inline void unmap_mapping_folio (struct folio *folio) { }static inline void mlock_new_folio (struct folio *folio) { }static inline bool need_mlock_drain (int cpu) {return false; }static inline void mlock_drain_local (void ) { }
mm/memory.c +2 -0
@@ -78,6 +78,7 @@#include <linux/sched/sysctl.h> #include <linux/pgalloc.h> #include <linux/uaccess.h> +#include <linux/kvm_types.h> #include <trace/events/kmem.h>
@@ -4244,6 +4245,7 @@ void unmap_mapping_folio(struct folio *folio) last_index, &details);i_mmap_unlock_read (mapping); } +EXPORT_SYMBOL_FOR_KVM (unmap_mapping_folio);/** * unmap_mapping_pages() - Unmap pages from processes.
Signature · 1 line
2.53.0.345.g96ddfc5eaa-goog
RFC PATCH 6/10 mm: filemap: Export filemap_remove_folio()
In reply to Ackerley TngExport filemap_remove_folio() for use by KVM.
KVM requires this function to remove folios from guest_memfd's filemap.
guest_memfd used to rely on higher-level, exported truncation functions. To track memory that is actually allocated (i.e. i_blocks, st_blocks), guest_memfd will need a custom truncation function to do accounting cleanup and will be using filemap_remove_folio() for truncation.
--- mm/filemap.c | 2 ++ 1 file changed, 2 insertions(+)
mm/filemap.c +2 -0
@@ -49,6 +49,7 @@#include <linux/sched/mm.h> #include <linux/sysctl.h> #include <linux/pgalloc.h> +#include <linux/kvm_types.h> #include <asm/tlbflush.h> #include "internal.h"
@@ -262,6 +263,7 @@ void filemap_remove_folio(struct folio *folio)filemap_free_folio (mapping, folio); } +EXPORT_SYMBOL_FOR_KVM (filemap_remove_folio);/* * page_cache_delete_batch - delete several folios from page cache
Signature · 1 line
2.53.0.345.g96ddfc5eaa-goog
RFC PATCH 7/10 KVM: guest_memfd: Implement custom truncation function
In reply to Ackerley TngImplement custom truncation function for guest_memfd, and replace calls to truncate_inode_pages_range() with calls to this custom truncation function.
The custom truncation function removes a lot of the generality supported by truncate_inode_pages_range() not required by guest_memfd, such as
+ sub-PAGE_SIZE truncations
+ Support for writeback
In a later patch, guest_memfd use this custom truncation function to handle updating of i_blocks and i_bytes in the inode during truncation.
--- virt/kvm/guest_memfd.c | 43 ++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 41 insertions(+), 2 deletions(-)
virt/kvm/guest_memfd.c +41 -2
@@ -247,6 +247,45 @@ static void kvm_gmem_invalidate_end(struct inode *inode, pgoff_t start,__kvm_gmem_invalidate_end (f, start, end); } +static void kvm_gmem_truncate_folio (struct folio *folio) +{ +folio_lock (folio); + +if (folio_mapped (folio)) +unmap_mapping_folio (folio); + +/* +* guest_memfd doesn't need writeback, skip anything to do with +* writeback and just clear the dirty flag. +*/ +folio_clear_dirty (folio); +filemap_remove_folio (folio); + +folio_unlock (folio); +} + +static void kvm_gmem_truncate_range (struct inode *inode,pgoff_t start, +size_t nr_pages) + +{ +struct folio_batch fbatch; +pgoff_t next; +pgoff_t last; +int i; + + last = start + nr_pages -1 ; + +folio_batch_init (&fbatch); + next = start; +while (filemap_get_folios (inode->i_mapping, &next, last, &fbatch)) { +for (i =0 ; i <folio_batch_count (&fbatch); ++i) +kvm_gmem_truncate_folio (fbatch.folios[i]); + +folio_batch_release (&fbatch); +cond_resched (); + } +} +static long kvm_gmem_punch_hole (struct inode *inode,loff_t offset,loff_t len) {pgoff_t start = offset >>PAGE_SHIFT ;
@@ -260,7 +299,7 @@ static long kvm_gmem_punch_hole(struct inode *inode, loff_t offset, loff_t len)kvm_gmem_invalidate_begin (inode, start, end); -truncate_inode_pages_range (inode->i_mapping, offset, offset + len -1 ); +kvm_gmem_truncate_range (inode, offset, len >>PAGE_SHIFT );kvm_gmem_invalidate_end (inode, start, end);
@@ -984,7 +1023,7 @@ static void kvm_gmem_evict_inode(struct inode *inode)truncate_inode_pages_final_prepare (mapping); -truncate_inode_pages_range (mapping,0 , inode->i_size); +kvm_gmem_truncate_range (inode,0 , inode->i_size >>PAGE_SHIFT );clear_inode (inode); }
Signature · 1 line
2.53.0.345.g96ddfc5eaa-goog
RFC PATCH 8/10 KVM: guest_memfd: Track amount of memory allocated on inode
In reply to Ackerley TngThe guest memfd currently does not update the inode's i_blocks and i_bytes count when memory is allocated or freed. Hence, st_blocks returned from fstat() is always 0.
Introduce byte accounting for guest memfd inodes. When a new folio is added to the filemap, add the folio's size using inode_add_bytes(). Conversely, when folios are truncated and removed from the mapping, sum their sizes and subtract the total from the inode's byte count via inode_sub_bytes().
With this change, stat.st_blocks for a guest_memfd will correctly report the number of 512-byte blocks allocated to the file, consistent with other memory-based filesystems like tmpfs.
--- virt/kvm/guest_memfd.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-)
virt/kvm/guest_memfd.c +13 -2
@@ -137,6 +137,8 @@ static struct folio *__kvm_gmem_get_folio(struct inode *inode, pgoff_t index)return ERR_PTR (ret); } +inode_add_bytes (inode,folio_size (folio)); +return folio; }
@@ -247,10 +249,14 @@ static void kvm_gmem_invalidate_end(struct inode *inode, pgoff_t start,__kvm_gmem_invalidate_end (f, start, end); } -static void kvm_gmem_truncate_folio (struct folio *folio) +static size_t kvm_gmem_truncate_folio (struct folio *folio) { +size_t nr_bytes; +folio_lock (folio); + nr_bytes =folio_size (folio); +if (folio_mapped (folio))unmap_mapping_folio (folio);
@@ -262,6 +268,8 @@ static void kvm_gmem_truncate_folio(struct folio *folio)filemap_remove_folio (folio);folio_unlock (folio); + +return nr_bytes; } static void kvm_gmem_truncate_range(struct inode *inode, pgoff_t start,
@@ -269,6 +277,7 @@ static void kvm_gmem_truncate_range(struct inode *inode, pgoff_t start, {struct folio_batch fbatch; +size_t nr_bytes =0 ;pgoff_t next;pgoff_t last;int i;
@@ -279,11 +288,13 @@ static void kvm_gmem_truncate_range(struct inode *inode, pgoff_t start, next = start;while (filemap_get_folios (inode->i_mapping, &next, last, &fbatch)) {for (i =0 ; i <folio_batch_count (&fbatch); ++i) -kvm_gmem_truncate_folio (fbatch.folios[i]); + nr_bytes +=kvm_gmem_truncate_folio (fbatch.folios[i]);folio_batch_release (&fbatch);cond_resched (); } + +inode_sub_bytes (inode, nr_bytes); }static long kvm_gmem_punch_hole (struct inode *inode,loff_t offset,loff_t len)
Signature · 1 line
2.53.0.345.g96ddfc5eaa-goog
RFC PATCH 9/10 KVM: selftests: Wrap fstat() to assert success
In reply to Ackerley TngExtend kvm_syscalls.h to wrap fstat() to assert success. This will be used in the next patch.
--- tools/testing/selftests/kvm/guest_memfd_test.c | 15 +++++---------- .../testing/selftests/kvm/include/kvm_syscalls.h | 2 ++ 2 files changed, 7 insertions(+), 10 deletions(-)
tools/testing/selftests/kvm/guest_memfd_test.c +5 -10
@@ -212,10 +212,8 @@ static void test_mmap_not_supported(int fd, size_t total_size)static void test_file_size (int fd,size_t total_size) {struct stat sb; -int ret; - ret =fstat (fd, &sb); -TEST_ASSERT (!ret,"fstat should succeed" ); +kvm_fstat (fd, &sb);TEST_ASSERT_EQ (sb.st_size, total_size);TEST_ASSERT_EQ (sb.st_blksize, page_size); }
@@ -303,25 +301,22 @@ static void test_create_guest_memfd_invalid_sizes(struct kvm_vm *vm,static void test_create_guest_memfd_multiple (struct kvm_vm *vm) { -int fd1, fd2, ret; +int fd1, fd2;struct stat st1, st2; fd1 =__vm_create_guest_memfd (vm, page_size,0 );TEST_ASSERT (fd1 !=-1 ,"memfd creation should succeed" ); - ret =fstat (fd1, &st1); -TEST_ASSERT (ret !=-1 ,"memfd fstat should succeed" ); +kvm_fstat (fd1, &st1);TEST_ASSERT (st1.st_size == page_size,"memfd st_size should match requested size" ); fd2 =__vm_create_guest_memfd (vm, page_size *2 ,0 );TEST_ASSERT (fd2 !=-1 ,"memfd creation should succeed" ); - ret =fstat (fd2, &st2); -TEST_ASSERT (ret !=-1 ,"memfd fstat should succeed" ); +kvm_fstat (fd2, &st2);TEST_ASSERT (st2.st_size == page_size *2 ,"second memfd st_size should match requested size" ); - ret =fstat (fd1, &st1); -TEST_ASSERT (ret !=-1 ,"memfd fstat should succeed" ); +kvm_fstat (fd1, &st1);TEST_ASSERT (st1.st_size == page_size,"first memfd st_size should still match requested size" );TEST_ASSERT (st1.st_ino != st2.st_ino,"different memfd should have different inode numbers" );
tools/testing/selftests/kvm/include/kvm_syscalls.h +2 -0
@@ -2,6 +2,7 @@#ifndef SELFTEST_KVM_SYSCALLS_H #define SELFTEST_KVM_SYSCALLS_H +#include <sys/stat.h> #include <sys/syscall.h> #define MAP_ARGS0 (m,...)
@@ -77,5 +78,6 @@ __KVM_SYSCALL_DEFINE(munmap, 2, void *, mem, size_t, size);__KVM_SYSCALL_DEFINE (close,1 , int, fd);__KVM_SYSCALL_DEFINE (fallocate,4 , int, fd, int, mode, loff_t, offset, loff_t, len); __KVM_SYSCALL_DEFINE(ftruncate, 2, unsigned int, fd, off_t, length); +__KVM_SYSCALL_DEFINE(fstat, 2, int, fd, struct stat *, buf);#endif /* SELFTEST_KVM_SYSCALLS_H */
Signature · 1 line
2.53.0.345.g96ddfc5eaa-goog
RFC PATCH 10/10 KVM: selftests: Test that st_blocks is updated on allocation
In reply to Ackerley TngThe st_blocks field reported by fstat should reflect the number of allocated 512-byte blocks for the guest memfd file.
Extend the fallocate test to verify that st_blocks is correctly updated when memory is allocated or deallocated via fallocate(FALLOC_FL_PUNCH_HOLE).
Add checks after each fallocate call to ensure that st_blocks increases on allocation, decreases when a hole is punched, and is restored when the hole is re-allocated. Also verify that st_blocks remains unchanged for failing fallocate calls.
--- tools/testing/selftests/kvm/guest_memfd_test.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
tools/testing/selftests/kvm/guest_memfd_test.c +17 -0
@@ -218,41 +218,58 @@ static void test_file_size(int fd, size_t total_size)TEST_ASSERT_EQ (sb.st_blksize, page_size); } +static void assert_st_blocks_matches_size (int fd,size_t expected_size) +{ +struct stat sb; + +kvm_fstat (fd, &sb); +TEST_ASSERT_EQ (sb.st_blocks, expected_size /512 ); +} +static void test_fallocate (int fd,size_t total_size) {int ret; ret =fallocate (fd,FALLOC_FL_KEEP_SIZE ,0 , total_size);TEST_ASSERT (!ret,"fallocate with aligned offset and size should succeed" ); +assert_st_blocks_matches_size (fd, total_size); ret =fallocate (fd,FALLOC_FL_KEEP_SIZE |FALLOC_FL_PUNCH_HOLE , page_size -1 , page_size);TEST_ASSERT (ret,"fallocate with unaligned offset should fail" ); +assert_st_blocks_matches_size (fd, total_size); ret =fallocate (fd,FALLOC_FL_KEEP_SIZE , total_size, page_size);TEST_ASSERT (ret,"fallocate beginning at total_size should fail" ); +assert_st_blocks_matches_size (fd, total_size); ret =fallocate (fd,FALLOC_FL_KEEP_SIZE , total_size + page_size, page_size);TEST_ASSERT (ret,"fallocate beginning after total_size should fail" ); +assert_st_blocks_matches_size (fd, total_size); ret =fallocate (fd,FALLOC_FL_KEEP_SIZE |FALLOC_FL_PUNCH_HOLE , total_size, page_size);TEST_ASSERT (!ret,"fallocate(PUNCH_HOLE) at total_size should succeed" ); +assert_st_blocks_matches_size (fd, total_size); ret =fallocate (fd,FALLOC_FL_KEEP_SIZE |FALLOC_FL_PUNCH_HOLE , total_size + page_size, page_size);TEST_ASSERT (!ret,"fallocate(PUNCH_HOLE) after total_size should succeed" ); +assert_st_blocks_matches_size (fd, total_size); ret =fallocate (fd,FALLOC_FL_KEEP_SIZE |FALLOC_FL_PUNCH_HOLE , page_size, page_size -1 );TEST_ASSERT (ret,"fallocate with unaligned size should fail" ); +assert_st_blocks_matches_size (fd, total_size); ret =fallocate (fd,FALLOC_FL_KEEP_SIZE |FALLOC_FL_PUNCH_HOLE , page_size, page_size);TEST_ASSERT (!ret,"fallocate(PUNCH_HOLE) with aligned offset and size should succeed" ); +assert_st_blocks_matches_size (fd, total_size - page_size); ret =fallocate (fd,FALLOC_FL_KEEP_SIZE , page_size, page_size);TEST_ASSERT (!ret,"fallocate to restore punched hole should succeed" ); +assert_st_blocks_matches_size (fd, total_size); }static void test_invalid_punch_hole (int fd,size_t total_size)
Signature · 1 line
2.53.0.345.g96ddfc5eaa-goog
David Hildenbrand (Arm)
maintainerQuoted 17 lines · Ackerley Tng
Hi,
Currently,
guest_memfddoesn't update inode'si_blocksori_bytesat all. Hence,st_blocksin the struct populated by a userspacefstat()call on aguest_memfdwill always be 0. This patch series makesguest_memfdtrack the amount of memory allocated on an inode, which allowsfstat()to accurately report that on requests from userspace.The inode's
i_blocksandi_bytesfields are updated when the folio is associated or disassociated from theguest_memfdinode, which are at allocation and truncation times respectively.To update inode fields at truncation time, this series implements a custom truncation function for
guest_memfd. An alternative would be to updatetruncate_inode_pages_range()to return the number of bytes truncated or add/use some hook.
Quoted 5 lines
Implementing a custom truncation function was chosen to provide flexibility for handling truncations in future when
guest_memfdsupports sources of pages other than the buddy allocator. This approach of a custom truncation function also aligns with shmem, which has a customshmem_truncate_range().
Just wondered how shmem does it: it's through dquot_alloc_block_nodirty() / dquot_free_block_nodirty().
It's a shame we can't just use folio_free(). Could we maybe have a different callback (when the mapping is still guaranteed to be around) from where we could update i_blocks on the freeing path?
Signature · 3 lines
Cheers, David
Quoted 24 lines · David Hildenbrand (Arm)
Quoted 17 lines · Ackerley Tng
Hi,
Currently,
guest_memfddoesn't update inode'si_blocksori_bytesat all. Hence,st_blocksin the struct populated by a userspacefstat()call on aguest_memfdwill always be 0. This patch series makesguest_memfdtrack the amount of memory allocated on an inode, which allowsfstat()to accurately report that on requests from userspace.The inode's
i_blocksandi_bytesfields are updated when the folio is associated or disassociated from theguest_memfdinode, which are at allocation and truncation times respectively.To update inode fields at truncation time, this series implements a custom truncation function for
guest_memfd. An alternative would be to updatetruncate_inode_pages_range()to return the number of bytes truncated or add/use some hook.Quoted 5 lines
Implementing a custom truncation function was chosen to provide flexibility for handling truncations in future when
guest_memfdsupports sources of pages other than the buddy allocator. This approach of a custom truncation function also aligns with shmem, which has a customshmem_truncate_range().
Quoted 4 lines
Just wondered how shmem does it: it's through
dquot_alloc_block_nodirty()/dquot_free_block_nodirty().It's a shame we can't just use
folio_free().
Yup, Hugh pointed out that struct address_space *mapping (and inode) may already have been freed by the time .free_folio() is called [1].
[1] https://lore.kernel.org/all/[email protected]/
Quoted 3 lines
Could we maybe have a
different callback (when the mapping is still guaranteed to be around) from where we could updatei_blockson the freeing path?
Do you mean that we should add a new callback to struct address_space_operations?
.invalidate_folio semantically seems suitable. This is called from truncate_cleanup_folio() and is conditioned on folio_needs_release(). guest_memfd could make itself need release, but IIUC that would cause a NULL pointer dereference in filemap_release_folio() since try_to_free_buffers() -> drop_buffers() will dereference folio->private.
From the name, .release_folio sounds eligible, but this is meant for releasing data attached to a folio, not quite the same as updating inode fields. This is also not called in the truncation path.
Quoted 5 lines
--
Cheers,David
David Hildenbrand (Arm)
maintainerQuoted 34 lines · Ackerley Tng
Quoted 24 lines · David Hildenbrand (Arm)
Quoted 17 lines · Ackerley Tng
Hi,
Currently,
guest_memfddoesn't update inode'si_blocksori_bytesat all. Hence,st_blocksin the struct populated by a userspacefstat()call on aguest_memfdwill always be 0. This patch series makesguest_memfdtrack the amount of memory allocated on an inode, which allowsfstat()to accurately report that on requests from userspace.The inode's
i_blocksandi_bytesfields are updated when the folio is associated or disassociated from theguest_memfdinode, which are at allocation and truncation times respectively.To update inode fields at truncation time, this series implements a custom truncation function for
guest_memfd. An alternative would be to updatetruncate_inode_pages_range()to return the number of bytes truncated or add/use some hook.Quoted 5 lines
Implementing a custom truncation function was chosen to provide flexibility for handling truncations in future when
guest_memfdsupports sources of pages other than the buddy allocator. This approach of a custom truncation function also aligns with shmem, which has a customshmem_truncate_range().Quoted 4 lines
Just wondered how shmem does it: it's through
dquot_alloc_block_nodirty()/dquot_free_block_nodirty().It's a shame we can't just use
folio_free().Yup, Hugh pointed out that struct
address_space*mapping (and inode) may already have been freed by the time .free_folio() is called [1].
Quoted 8 lines
[1] https://lore.kernel.org/all/[email protected]/
Quoted 3 lines
Could we maybe have a
different callback (when the mapping is still guaranteed to be around) from where we could updatei_blockson the freeing path?Do you mean that we should add a new callback to struct
address_space_operations?
If that avoids having to implement truncation completely ourselves, that might be one option we could discuss, yes.
Something like:
Documentation/filesystems/vfs.rst +6 -0
@@ -764,6 +764,7 @@ cache in your filesystem. The following members are defined: sector_t (*bmap)(struct address_space *, sector_t); void (*invalidate_folio) (struct folio *, size_t start, size_t len); bool (*release_folio)(struct folio *, gfp_t); + void (*remove_folio)(struct folio *folio); void (*free_folio)(struct folio *); ssize_t (*direct_IO)(struct kiocb *, struct iov_iter *iter); int (*migrate_folio)(struct mapping *, struct folio *dst,
@@ -922,6 +923,11 @@ cache in your filesystem. The following members are defined: its release_folio will need to ensure this. Possibly it can clear the uptodate flag if it cannot free private data yet. +``remove_folio`` + remove_folio is called just before the folio is removed from the + page cache in order to allow the cleanup of properties (e.g., + accounting) that needs the address_space mapping. + ``free_folio`` free_folio is called once the folio is no longer visible in the page cache in order to allow the cleanup of any private data.
include/linux/fs.h +1 -0
@@ -422,6 +422,7 @@ struct address_space_operations {sector_t (*bmap)(struct address_space *,sector_t );void (*invalidate_folio) (struct folio *,size_t offset,size_t len);bool (*release_folio)(struct folio *,gfp_t ); +void (*remove_folio)(struct folio *folio);void (*free_folio)(struct folio *folio);ssize_t (*direct_IO)(struct kiocb *,struct iov_iter *iter);/*
mm/filemap.c +6 -0
@@ -250,8 +250,14 @@ void filemap_free_folio(struct address_space *mapping, struct folio *folio)void filemap_remove_folio (struct folio *folio) {struct address_space *mapping = folio->mapping; +void (*remove_folio)(struct folio *);BUG_ON (!folio_test_locked (folio)); + + remove_folio = mapping->a_ops->remove_folio; +if (unlikely (remove_folio)) +remove_folio (folio); +spin_lock (&mapping->host->i_lock);xa_lock_irq (&mapping->i_pages);__filemap_remove_folio (folio,NULL );
Ideally we'd perform it under the lock just after clearing folio->mapping, but I guess that might be more controversial.
For accounting you need the above might be good enough, but I am not sure for how many other use cases there might be.
Signature · 3 lines
Cheers, David
Quoted 61 lines · David Hildenbrand (Arm)
[...snip...]
Quoted 6 lines
Quoted 3 lines
Could we maybe have a
different callback (when the mapping is still guaranteed to be around) from where we could updatei_blockson the freeing path?Do you mean that we should add a new callback to struct
address_space_operations?If that avoids having to implement truncation completely ourselves, that might be one option we could discuss, yes.
Something like:
Documentation/filesystems/vfs.rst +6
-0@@ -764,6 +764,7 @@ cache in your filesystem. The following members are defined: sector_t (*bmap)(struct address_space *, sector_t); void (*invalidate_folio) (struct folio *, size_t start, size_t len); bool (*release_folio)(struct folio *, gfp_t); + void (*remove_folio)(struct folio *folio); void (*free_folio)(struct folio *); ssize_t (*direct_IO)(struct kiocb *, struct iov_iter *iter); int (*migrate_folio)(struct mapping *, struct folio *dst,@@ -922,6 +923,11 @@ cache in your filesystem. The following members are defined: its release_folio will need to ensure this. Possibly it can clear the uptodate flag if it cannot free private data yet. +``remove_folio`` + remove_folio is called just before the folio is removed from the + page cache in order to allow the cleanup of properties (e.g., + accounting) that needs the address_space mapping. + ``free_folio`` free_folio is called once the folio is no longer visible in the page cache in order to allow the cleanup of any private data.include/linux/fs.h +1
-0@@ -422,6 +422,7 @@ struct address_space_operations {sector_t (*bmap)(struct address_space *,sector_t );void (*invalidate_folio) (struct folio *,size_t offset,size_t len);bool (*release_folio)(struct folio *,gfp_t ); +void (*remove_folio)(struct folio *folio);void (*free_folio)(struct folio *folio);ssize_t (*direct_IO)(struct kiocb *,struct iov_iter *iter);/* mm/filemap.c +1
-0@@ -250,8 +250,14 @@ void filemap_free_folio(struct address_space *mapping, struct folio *folio)void filemap_remove_folio (struct folio *folio) {struct address_space *mapping = folio->mapping; +void (*remove_folio)(struct folio *);
mm/filemap.c void filemap_free_folio(struct address_space *mapping, struct folio *folio) · line 263
Quoted 10 lines
mm/filemap.c +5
-0BUG_ON (!folio_test_locked (folio)); + + remove_folio = mapping->a_ops->remove_folio; +if (unlikely (remove_folio)) +remove_folio (folio); +spin_lock (&mapping->host->i_lock);xa_lock_irq (&mapping->i_pages);__filemap_remove_folio (folio,NULL );
Thanks for this suggestion, I'll try this out and send another revision.
Quoted 11 lines
Ideally we'd perform it under the lock just after clearing
folio->mapping, but I guess that might be more controversial.For accounting you need the above might be good enough, but I am not sure for how many other use cases there might be.
--
Cheers,David
Quoted 63 lines · Ackerley Tng
Quoted 61 lines · David Hildenbrand (Arm)
[...snip...]
Quoted 6 lines
Quoted 3 lines
Could we maybe have a
different callback (when the mapping is still guaranteed to be around) from where we could updatei_blockson the freeing path?Do you mean that we should add a new callback to struct
address_space_operations?If that avoids having to implement truncation completely ourselves, that might be one option we could discuss, yes.
Something like:
Documentation/filesystems/vfs.rst +6
-0@@ -764,6 +764,7 @@ cache in your filesystem. The following members are defined: sector_t (*bmap)(struct address_space *, sector_t); void (*invalidate_folio) (struct folio *, size_t start, size_t len); bool (*release_folio)(struct folio *, gfp_t); + void (*remove_folio)(struct folio *folio); void (*free_folio)(struct folio *); ssize_t (*direct_IO)(struct kiocb *, struct iov_iter *iter); int (*migrate_folio)(struct mapping *, struct folio *dst,@@ -922,6 +923,11 @@ cache in your filesystem. The following members are defined: its release_folio will need to ensure this. Possibly it can clear the uptodate flag if it cannot free private data yet. +``remove_folio`` + remove_folio is called just before the folio is removed from the + page cache in order to allow the cleanup of properties (e.g., + accounting) that needs the address_space mapping. + ``free_folio`` free_folio is called once the folio is no longer visible in the page cache in order to allow the cleanup of any private data.include/linux/fs.h +1
-0@@ -422,6 +422,7 @@ struct address_space_operations {sector_t (*bmap)(struct address_space *,sector_t );void (*invalidate_folio) (struct folio *,size_t offset,size_t len);bool (*release_folio)(struct folio *,gfp_t ); +void (*remove_folio)(struct folio *folio);void (*free_folio)(struct folio *folio);ssize_t (*direct_IO)(struct kiocb *,struct iov_iter *iter);/* mm/filemap.c +1
-0@@ -250,8 +250,14 @@ void filemap_free_folio(struct address_space *mapping, struct folio *folio)void filemap_remove_folio (struct folio *folio) {struct address_space *mapping = folio->mapping; +void (*remove_folio)(struct folio *);
Quoted 17 lines
mm/filemap.c void filemap_free_folio(struct address_space *mapping, struct folio *folio) · line 263
Quoted 10 lines
mm/filemap.c +5
-0BUG_ON (!folio_test_locked (folio)); + + remove_folio = mapping->a_ops->remove_folio; +if (unlikely (remove_folio)) +remove_folio (folio); +spin_lock (&mapping->host->i_lock);xa_lock_irq (&mapping->i_pages);__filemap_remove_folio (folio,NULL );Thanks for this suggestion, I'll try this out and send another revision.
Quoted 4 lines
Ideally we'd perform it under the lock just after clearing
folio->mapping, but I guess that might be more controversial.
I'm not sure which lock you were referring to, I hope it's not the inode's i_lock? Why is calling the callback under lock frowned upon?
I found .remove_folio also had to be called from delete_from_page_cache_batch() for it to work. Then I saw that both of those functions already use filemap_unaccount_folio(), and after all, like you said, guest_memfd will be using this callback for accounting, so in RFC v2 [1] I used .unaccount_folio instead, and it is called under the inode's i_lock from filemap_unaccount_folio().
[1] https://lore.kernel.org/all/[email protected]/T/
Quoted 7 lines
Quoted 7 lines
For accounting you need the above might be good enough, but I am not sure for how many other use cases there might be.
--
Cheers,David
David Hildenbrand (Arm)
maintainerQuoted 73 lines · Ackerley Tng
Quoted 71 lines · Ackerley Tng
Quoted 55 lines · David Hildenbrand (Arm)
[...snip...]
If that avoids having to implement truncation completely ourselves, that might be one option we could discuss, yes.
Something like:
Documentation/filesystems/vfs.rst +6
-0@@ -764,6 +764,7 @@ cache in your filesystem. The following members are defined: sector_t (*bmap)(struct address_space *, sector_t); void (*invalidate_folio) (struct folio *, size_t start, size_t len); bool (*release_folio)(struct folio *, gfp_t); + void (*remove_folio)(struct folio *folio); void (*free_folio)(struct folio *); ssize_t (*direct_IO)(struct kiocb *, struct iov_iter *iter); int (*migrate_folio)(struct mapping *, struct folio *dst,@@ -922,6 +923,11 @@ cache in your filesystem. The following members are defined: its release_folio will need to ensure this. Possibly it can clear the uptodate flag if it cannot free private data yet. +``remove_folio`` + remove_folio is called just before the folio is removed from the + page cache in order to allow the cleanup of properties (e.g., + accounting) that needs the address_space mapping. + ``free_folio`` free_folio is called once the folio is no longer visible in the page cache in order to allow the cleanup of any private data.include/linux/fs.h +1
-0@@ -422,6 +422,7 @@ struct address_space_operations {sector_t (*bmap)(struct address_space *,sector_t );void (*invalidate_folio) (struct folio *,size_t offset,size_t len);bool (*release_folio)(struct folio *,gfp_t ); +void (*remove_folio)(struct folio *folio);void (*free_folio)(struct folio *folio);ssize_t (*direct_IO)(struct kiocb *,struct iov_iter *iter);/* mm/filemap.c +1
-0@@ -250,8 +250,14 @@ void filemap_free_folio(struct address_space *mapping, struct folio *folio)void filemap_remove_folio (struct folio *folio) {struct address_space *mapping = folio->mapping; +void (*remove_folio)(struct folio *);mm/filemap.c void filemap_free_folio(struct address_space *mapping, struct folio *folio) · line 263
Quoted 10 lines
mm/filemap.c +5
-0BUG_ON (!folio_test_locked (folio)); + + remove_folio = mapping->a_ops->remove_folio; +if (unlikely (remove_folio)) +remove_folio (folio); +spin_lock (&mapping->host->i_lock);xa_lock_irq (&mapping->i_pages);__filemap_remove_folio (folio,NULL );Thanks for this suggestion, I'll try this out and send another revision.
Quoted 1 line
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Ideally we'd perform it under the lock just after clearing
folio->mapping, but I guess that might be more controversial.I'm not sure which lock you were referring to, I hope it's not the inode's
i_lock? Why is calling the callback under lock frowned upon?
I meant the two locks: mapping->host->i_lock and mapping->i_pages.
I'd assume new callbacks that might result in holding these precious locks longer might be a problem for some people. Well, maybe, maybe not.
I guess .free_folio() is called outside the lock because it's assumed to possibly do more expensive operations.
Signature · 3 lines
Cheers, David
Quoted 78 lines · David Hildenbrand (Arm)
Quoted 77 lines · Ackerley Tng
Quoted 74 lines · Ackerley Tng
Quoted 55 lines · David Hildenbrand (Arm)
[...snip...]
If that avoids having to implement truncation completely ourselves, that might be one option we could discuss, yes.
Something like:
Documentation/filesystems/vfs.rst +6
-0@@ -764,6 +764,7 @@ cache in your filesystem. The following members are defined: sector_t (*bmap)(struct address_space *, sector_t); void (*invalidate_folio) (struct folio *, size_t start, size_t len); bool (*release_folio)(struct folio *, gfp_t); + void (*remove_folio)(struct folio *folio); void (*free_folio)(struct folio *); ssize_t (*direct_IO)(struct kiocb *, struct iov_iter *iter); int (*migrate_folio)(struct mapping *, struct folio *dst,@@ -922,6 +923,11 @@ cache in your filesystem. The following members are defined: its release_folio will need to ensure this. Possibly it can clear the uptodate flag if it cannot free private data yet. +``remove_folio`` + remove_folio is called just before the folio is removed from the + page cache in order to allow the cleanup of properties (e.g., + accounting) that needs the address_space mapping. + ``free_folio`` free_folio is called once the folio is no longer visible in the page cache in order to allow the cleanup of any private data.include/linux/fs.h +1
-0@@ -422,6 +422,7 @@ struct address_space_operations {sector_t (*bmap)(struct address_space *,sector_t );void (*invalidate_folio) (struct folio *,size_t offset,size_t len);bool (*release_folio)(struct folio *,gfp_t ); +void (*remove_folio)(struct folio *folio);void (*free_folio)(struct folio *folio);ssize_t (*direct_IO)(struct kiocb *,struct iov_iter *iter);/* mm/filemap.c +1
-0@@ -250,8 +250,14 @@ void filemap_free_folio(struct address_space *mapping, struct folio *folio)void filemap_remove_folio (struct folio *folio) {struct address_space *mapping = folio->mapping; +void (*remove_folio)(struct folio *);mm/filemap.c void filemap_free_folio(struct address_space *mapping, struct folio *folio) · line 263
Quoted 10 lines
mm/filemap.c +5
-0BUG_ON (!folio_test_locked (folio)); + + remove_folio = mapping->a_ops->remove_folio; +if (unlikely (remove_folio)) +remove_folio (folio); +spin_lock (&mapping->host->i_lock);xa_lock_irq (&mapping->i_pages);__filemap_remove_folio (folio,NULL );Thanks for this suggestion, I'll try this out and send another revision.
Quoted 4 lines
Ideally we'd perform it under the lock just after clearing
folio->mapping, but I guess that might be more controversial.
Quoted 8 lines
Quoted 2 lines
I'm not sure which lock you were referring to, I hope it's not the inode's
i_lock? Why is calling the callback under lock frowned upon?I meant the two locks:
mapping->host->i_lockandmapping->i_pages.I'd assume new callbacks that might result in holding these precious locks longer might be a problem for some people. Well, maybe, maybe not.
The extra time (for guest_memfd, and almost no extra time for other filesystems) is on the truncation path, hopefully that isn't a hot path!
Quoted 3 lines
I guess .free_folio() is called outside the lock because it's assumed to possibly do more expensive operations.
I thought .free_folio() was called outside of the lock because after the folio is removed from the filemap, there should be no more inode/filemap related contention, so any cleanup can definitely be done outside the inode/filemap locks.
Quoted 4 lines
--
Cheers,David