mm: filemap: add filemap_grab_folios
Subsystems (5)
cover mm: filemap: add filemap_grab_folios
Based on David's suggestion for speeding up guest_memfd memory population [1] made at the guest_memfd upstream call on 5 Dec 2024 [2], this adds filemap_grab_folios that grabs multiple folios at a time.
Motivation
When profiling guest_memfd population and comparing the results with population of anonymous memory via UFFDIO_COPY, I observed that the former was up to 20% slower, mainly due to adding newly allocated pages to the pagecache. As far as I can see, the two main contributors to it are pagecache locking and tree traversals needed for every folio. The RFC attempts to partially mitigate those by adding multiple folios at a time to the pagecache.
Testing
With the change applied, I was able to observe a 10.3% (708 to 635 ms) speedup in a selftest that populated 3GiB guest_memfd and a 9.5% (990 to 904 ms) speedup when restoring a 3GiB guest_memfd VM snapshot using a custom Firecracker version, both on Intel Ice Lake.
Limitations
While filemap_grab_folios handles THP/large folios internally and deals with reclaim artifacts in the pagecache (shadows), for simplicity reasons, the RFC does not support those as it demonstrates the optimisation applied to guest_memfd, which only uses small folios and does not support reclaim at the moment.
Implementation
I am aware of existing filemap APIs operating on folio batches, however I was not able to find one for the use case in question. I was also thinking about making use of the folio_batch struct, but was not able to convince myself that it was useful. Instead, a plain array of folio pointers is allocated on stack and passed down the callchain. A bitmap is used to keep track of indexes whose folios were already present in the pagecache to prevent allocations. This does not look very clean to me and I am more than open to hearing about better approaches.
Not being an expert in xarray, I do not know an idiomatic way to advance the index if xas_next is called directly after instantiation of the state that was never walked, so I used a call to xas_set.
While the series focuses on optimising _adding_ folios to the pagecache, I was also experimenting with batching of pagecache _querying_. Specifically, I tried to make use of filemap_get_folios instead of filemap_get_entry, but I could not observe any visible speedup.
The series is applied on top of [1] as the 1st patch implements filemap_grab_folios, while the 2nd patch makes use of it in the guest_memfd's write syscall as a first user.
Questions:
- Does the approach look reasonable in general?
- Can the API be kept specialised to the non-reclaim-supported case or does it need to be generic?
- Would it be sensible to add a specialised small-folio-only version of filemap_grab_folios at the beginning and extend it to large folios
later on?
- Are there better ways to implement batching or even achieve the optimisaton goal in another way?
[1]: https://lore.kernel.org/kvm/[email protected]/T/
[2]: https://docs.google.com/document/d/1M6766BzdY1Lhk7LiR5IqVR8B8mG3cr-cxTxOrAosPOk/edit?tab=t.0
Thanks
Nikita
Nikita Kalyazin (2):
mm: filemap: add filemap_grab_folios
KVM: guest_memfd: use filemap_grab_folios in write
include/linux/pagemap.h | 31 +++++ mm/filemap.c | 263 ++++++++++++++++++++++++++++++++++++++++ virt/kvm/guest_memfd.c | 176 ++++++++++++++++++++++----- 3 files changed, 437 insertions(+), 33 deletions(-)
base-commit: 643cff38ebe84c39fbd5a0fc3ab053cd941b9f94
Signature · 1 line
2.40.1
RFC PATCH 1/2 mm: filemap: add filemap_grab_folios
Similar to filemap_grab_folio, but grabs multiple folios at a time. filemap_grab_folios attempts to get 128 adjacent folios in the pagecache starting at the specified index. Whenever a folio is not found, it allocates a new one and adds it to the pagecache.
The following is not currently supported:
- large folios
- reclaim effects in the pagecache (shadows)
An equivalent of the following callstack is implemented to work on multiple folios:
- filemap_grab_folio
- __filemap_get_folio
- filemap_add_folio
- __filemap_add_folio
--- include/linux/pagemap.h | 31 +++++ mm/filemap.c | 263 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 294 insertions(+)
include/linux/pagemap.h +31 -0
@@ -747,9 +747,16 @@ static inline fgf_t fgf_set_order(size_t size)return (__force fgf_t)((shift -PAGE_SHIFT ) <<26 ); } +/** +* Folio batch size used by __filemap_get_folios. +*/ +#define FILEMAP_GET_FOLIOS_BATCH_SIZE 128 +void *filemap_get_entry (struct address_space *mapping,pgoff_t index);struct folio *__filemap_get_folio (struct address_space *mapping,pgoff_t index,fgf_t fgp_flags,gfp_t gfp); +int __filemap_get_folios (struct address_space *mapping,pgoff_t index, +fgf_t fgp_flags,gfp_t gfp,struct folio **folios,int num);struct page *pagecache_get_page (struct address_space *mapping,pgoff_t index,fgf_t fgp_flags,gfp_t gfp);
@@ -808,6 +815,30 @@ static inline struct folio *filemap_grab_folio(struct address_space *mapping, mapping_gfp_mask(mapping)); } +/** +* filemap_grab_folios - grab folios from the page cache +* @mapping: The address space to search +* @index: The page index to start with +* @folios: Output buffer for found or created folios +* @num: Number of folios to grab +* +* Looks up @num page cache entries at @mapping starting from @index. If no +* folio is found at the index, a new folio is created. The folios are locked, +* and marked as accessed. +* +* Return: The total number of found and created folios. Returned folios will +* always have adjacent indexes starting from @index. If no folios are found +* and created, -ENOMEM is returned. +*/ +static inline int filemap_grab_folios (struct address_space *mapping, +pgoff_t index,struct folio **folios, +int num) +{ +return __filemap_get_folios (mapping, index, +FGP_LOCK |FGP_ACCESSED |FGP_CREAT , +mapping_gfp_mask (mapping), folios, num); +} +/** * find_get_page - find and get a page reference * @mapping: the address_space to search
mm/filemap.c +263 -0
281 lines not on this page · raw message
Signature · 1 line
2.40.1
RFC PATCH 2/2 KVM: guest_memfd: use filemap_grab_folios in write
In reply to Nikita KalyazinThe write syscall on guest_memfd makes use of filemap_grab_folios to grab folios in batches. This speeds up population by 8.3% due to the reduction in locking and tree walking when adding folios to the pagecache.
--- virt/kvm/guest_memfd.c | 176 +++++++++++++++++++++++++++++++++-------- 1 file changed, 143 insertions(+), 33 deletions(-)
virt/kvm/guest_memfd.c +143 -33
@@ -102,17 +102,134 @@ static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index)return filemap_grab_folio (inode->i_mapping, index); } +/* +* Returns locked folios on success. The caller is responsible for +* setting the up-to-date flag before the memory is mapped into the guest. +* There is no backing storage for the memory, so the folios will remain +* up-to-date until they're removed. +* +* Ignore accessed, referenced, and dirty flags. The memory is +* unevictable and there is no storage to write back to. +*/ +static int kvm_gmem_get_folios (struct inode *inode,pgoff_t index, +struct folio **folios,int num) +{ +return filemap_grab_folios (inode->i_mapping, index, folios, num); +} +#if defined(CONFIG_KVM_GENERIC_PRIVATE_MEM ) && !defined(CONFIG_KVM_AMD_SEV ) +static int kvm_kmem_gmem_write_inner (struct inode *inode,pgoff_t index, +const void __user *buf, +struct folio **folios,int num) +{ +int ret, i, num_grabbed, num_written; + + num_grabbed =kvm_gmem_get_folios (inode, index, folios, num); +if (num_grabbed <0 ) +return num_grabbed; + +for (i =0 ; i < num_grabbed; i++) { +struct folio *folio = folios[i]; +void *vaddr; + +if (folio_test_hwpoison (folio)) { +folio_unlock (folio); +folio_put (folio); + ret = -EFAULT ; +break ; + } + +if (folio_test_uptodate (folio)) { +folio_unlock (folio); +folio_put (folio); + ret = -ENOSPC ; +break ; + } + +folio_unlock (folio); + + vaddr =kmap_local_folio (folio,0 ); + ret =copy_from_user (vaddr, buf + (i <<PAGE_SHIFT ),PAGE_SIZE ); +if (ret) + ret = -EINVAL ; +kunmap_local (vaddr); + +if (ret) { +folio_put (folio); +break ; + }else { +kvm_gmem_mark_prepared (folio); +folio_put (folio); + } + } + + num_written = i; + +for (i = num_written; i < num_grabbed; i++) { +folio_unlock (folios[i]); +folio_put (folios[i]); + } + +return num_written ?: ret; +} + +static struct folio *kvm_kmem_gmem_write_folio (struct inode *inode,pgoff_t index, +const char __user *buf) +{ +struct folio *folio; +void *vaddr; +int ret =0 ; + + folio =kvm_gmem_get_folio (inode, index); +if (IS_ERR (folio)) +return ERR_PTR (-EFAULT ); + +if (folio_test_hwpoison (folio)) { + ret = -EFAULT ; + goto out_unlock_put; + } + +if (folio_test_uptodate (folio)) { + ret = -ENOSPC ; + goto out_unlock_put; + } + +folio_unlock (folio); + + vaddr =kmap_local_folio (folio,0 ); + ret =copy_from_user (vaddr, buf,PAGE_SIZE ); +if (ret) + ret = -EINVAL ; +kunmap_local (vaddr); + +if (ret) { +folio_put (folio); +kvm_gmem_mark_prepared (folio); + goto out_err; + } + +folio_put (folio); + +return folio; + +out_unlock_put: +folio_unlock (folio); +folio_put (folio); +out_err: +return ERR_PTR (ret); +} +static ssize_t kvm_kmem_gmem_write (struct file *file,const char __user *buf,size_t count,loff_t *offset) { +struct inode *inode =file_inode (file); +int ret =0 , batch_size =FILEMAP_GET_FOLIOS_BATCH_SIZE ;pgoff_t start, end, index; -ssize_t ret =0 ;if (!PAGE_ALIGNED (*offset) || !PAGE_ALIGNED (count))return -EINVAL ; -if (*offset + count >i_size_read (file_inode (file))) +if (*offset + count >i_size_read (inode))return -EINVAL ;if (!buf)
@@ -123,9 +240,8 @@ static ssize_t kvm_kmem_gmem_write(struct file *file, const char __user *buf,filemap_invalidate_lock (file->f_mapping); -for (index = start; index < end; ) { -struct folio *folio; -void *vaddr; +for (index = start; index + batch_size -1 < end; ) { +struct folio *folios[FILEMAP_GET_FOLIOS_BATCH_SIZE ] = {NULL };pgoff_t buf_offset = (index - start) <<PAGE_SHIFT ;if (signal_pending (current)) {
@@ -133,46 +249,40 @@ static ssize_t kvm_kmem_gmem_write(struct file *file, const char __user *buf, goto out; } - folio =kvm_gmem_get_folio (file_inode (file), index); -if (IS_ERR (folio)) { - ret = -EFAULT ; + ret =kvm_kmem_gmem_write_inner (inode, index, buf + buf_offset, folios, batch_size); +if (ret <0 ) goto out; - } -if (folio_test_hwpoison (folio)) { -folio_unlock (folio); -folio_put (folio); - ret = -EFAULT ; + index += ret; +if (ret < batch_size) +break ; + } + +for (; index < end; index++) { +struct folio *folio; +pgoff_t buf_offset = (index - start) <<PAGE_SHIFT ; + +if (signal_pending (current)) { + ret = -EINTR ; goto out; } -if (folio_test_uptodate (folio)) { -folio_unlock (folio); -folio_put (folio); - ret = -ENOSPC ; + folio =kvm_kmem_gmem_write_folio (inode, index, + buf + buf_offset); +if (IS_ERR (folio)) { + ret =PTR_ERR (folio); goto out; } - -folio_unlock (folio); - - vaddr =kmap_local_folio (folio,0 ); - ret =copy_from_user (vaddr, buf + buf_offset,PAGE_SIZE ); -if (ret) - ret = -EINVAL ; -kunmap_local (vaddr); - -kvm_gmem_mark_prepared (folio); -folio_put (folio); - - index =folio_next_index (folio); - *offset += PAGE_SIZE; } out:filemap_invalidate_unlock (file->f_mapping); +if (index > start) { + *offset += (index - start) <<PAGE_SHIFT ; +return (index - start) <<PAGE_SHIFT ; + } -return ret && start == (*offset >>PAGE_SHIFT ) ? - ret : *offset - (start <<PAGE_SHIFT ); +return ret; }#endif
Signature · 1 line
2.40.1
Re: [RFC PATCH 2/2] KVM: guest_memfd: use filemap_grab_folios in write
Quoted 82 lines · Nikita Kalyazin
The write syscall on
guest_memfdmakes use offilemap_grab_foliosto grab folios in batches. This speeds up population by 8.3% due to the reduction in locking and tree walking when adding folios to the pagecache.Signed-off-by Nikita Kalyazin <[email protected]>--- virt/kvm/guest_memfd.c | 176 +++++++++++++++++++++++++++++++++-------- 1 file changed, 143 insertions(+), 33 deletions(-)virt/kvm/guest_memfd.c +63
-0@@ -102,17 +102,134 @@ static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index)return filemap_grab_folio (inode->i_mapping, index); } +/* +* Returns locked folios on success. The caller is responsible for +* setting the up-to-date flag before the memory is mapped into the guest. +* There is no backing storage for the memory, so the folios will remain +* up-to-date until they're removed. +* +* Ignore accessed, referenced, and dirty flags. The memory is +* unevictable and there is no storage to write back to. +*/ +static int kvm_gmem_get_folios (struct inode *inode,pgoff_t index, +struct folio **folios,int num) +{ +return filemap_grab_folios (inode->i_mapping, index, folios, num); +} +#if defined(CONFIG_KVM_GENERIC_PRIVATE_MEM ) && !defined(CONFIG_KVM_AMD_SEV ) +static int kvm_kmem_gmem_write_inner (struct inode *inode,pgoff_t index, +const void __user *buf, +struct folio **folios,int num) +{ +int ret, i, num_grabbed, num_written; + + num_grabbed =kvm_gmem_get_folios (inode, index, folios, num); +if (num_grabbed <0 ) +return num_grabbed; + +for (i =0 ; i < num_grabbed; i++) { +struct folio *folio = folios[i]; +void *vaddr; + +if (folio_test_hwpoison (folio)) { +folio_unlock (folio); +folio_put (folio); + ret = -EFAULT ; +break ; + } + +if (folio_test_uptodate (folio)) { +folio_unlock (folio); +folio_put (folio); + ret = -ENOSPC ; +break ; + } + +folio_unlock (folio); + + vaddr =kmap_local_folio (folio,0 ); + ret =copy_from_user (vaddr, buf + (i <<PAGE_SHIFT ),PAGE_SIZE ); +if (ret) + ret = -EINVAL ; +kunmap_local (vaddr); + +if (ret) { +folio_put (folio); +break ; + }else { +kvm_gmem_mark_prepared (folio); +folio_put (folio); + } + } + + num_written = i; + +for (i = num_written; i < num_grabbed; i++) {
virt/kvm/guest_memfd.c static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index) · line 178
Quoted 10 lines
virt/kvm/guest_memfd.c +9
-0+folio_unlock (folios[i]); +folio_put (folios[i]); + } + +return num_written ?: ret; +} + +static struct folio *kvm_kmem_gmem_write_folio (struct inode *inode,pgoff_t index, +const char __user *buf)
This could probably be rewritten as:
struct folio *p_folio; int ret; ret = kvm_kmem_gmem_write_inner(inode, index, buf, &p_folio, 1); if (ret == 1) return p_folio; else return ERR_PTR(ret);
Would remove a few lines of duplicated code and use only one prototype.
Mike
+{
Quoted 140 lines
virt/kvm/guest_memfd.c +70
-33+struct folio *folio; +void *vaddr; +int ret =0 ; + + folio =kvm_gmem_get_folio (inode, index); +if (IS_ERR (folio)) +return ERR_PTR (-EFAULT ); + +if (folio_test_hwpoison (folio)) { + ret = -EFAULT ; + goto out_unlock_put; + } + +if (folio_test_uptodate (folio)) { + ret = -ENOSPC ; + goto out_unlock_put; + } + +folio_unlock (folio); + + vaddr =kmap_local_folio (folio,0 ); + ret =copy_from_user (vaddr, buf,PAGE_SIZE ); +if (ret) + ret = -EINVAL ; +kunmap_local (vaddr); + +if (ret) { +folio_put (folio); +kvm_gmem_mark_prepared (folio); + goto out_err; + } + +folio_put (folio); + +return folio; + +out_unlock_put: +folio_unlock (folio); +folio_put (folio); +out_err: +return ERR_PTR (ret); +} +static ssize_t kvm_kmem_gmem_write (struct file *file,const char __user *buf,size_t count,loff_t *offset) { +struct inode *inode =file_inode (file); +int ret =0 , batch_size =FILEMAP_GET_FOLIOS_BATCH_SIZE ;pgoff_t start, end, index; -ssize_t ret =0 ;if (!PAGE_ALIGNED (*offset) || !PAGE_ALIGNED (count))return -EINVAL ; -if (*offset + count >i_size_read (file_inode (file))) +if (*offset + count >i_size_read (inode))return -EINVAL ;if (!buf)@@ -123,9 +240,8 @@ static ssize_t kvm_kmem_gmem_write(struct file *file, const char __user *buf,filemap_invalidate_lock (file->f_mapping); -for (index = start; index < end; ) { -struct folio *folio; -void *vaddr; +for (index = start; index + batch_size -1 < end; ) { +struct folio *folios[FILEMAP_GET_FOLIOS_BATCH_SIZE ] = {NULL };pgoff_t buf_offset = (index - start) <<PAGE_SHIFT ;if (signal_pending (current)) {@@ -133,46 +249,40 @@ static ssize_t kvm_kmem_gmem_write(struct file *file, const char __user *buf, goto out; } - folio =kvm_gmem_get_folio (file_inode (file), index); -if (IS_ERR (folio)) { - ret = -EFAULT ; + ret =kvm_kmem_gmem_write_inner (inode, index, buf + buf_offset, folios, batch_size); +if (ret <0 ) goto out; - } -if (folio_test_hwpoison (folio)) { -folio_unlock (folio); -folio_put (folio); - ret = -EFAULT ; + index += ret; +if (ret < batch_size) +break ; + } + +for (; index < end; index++) { +struct folio *folio; +pgoff_t buf_offset = (index - start) <<PAGE_SHIFT ; + +if (signal_pending (current)) { + ret = -EINTR ; goto out; } -if (folio_test_uptodate (folio)) { -folio_unlock (folio); -folio_put (folio); - ret = -ENOSPC ; + folio =kvm_kmem_gmem_write_folio (inode, index, + buf + buf_offset); +if (IS_ERR (folio)) { + ret =PTR_ERR (folio); goto out; } - -folio_unlock (folio); - - vaddr =kmap_local_folio (folio,0 ); - ret =copy_from_user (vaddr, buf + buf_offset,PAGE_SIZE ); -if (ret) - ret = -EINVAL ; -kunmap_local (vaddr); - -kvm_gmem_mark_prepared (folio); -folio_put (folio); - - index =folio_next_index (folio); - *offset += PAGE_SIZE; } out:filemap_invalidate_unlock (file->f_mapping); +if (index > start) { + *offset += (index - start) <<PAGE_SHIFT ; +return (index - start) <<PAGE_SHIFT ; + } -return ret && start == (*offset >>PAGE_SHIFT ) ? - ret : *offset - (start <<PAGE_SHIFT ); +return ret; }#endif
Re: [RFC PATCH 2/2] KVM: guest_memfd: use filemap_grab_folios in write
Quoted 103 lines · Mike Day
Quoted 93 lines · Nikita Kalyazin
The write syscall on
guest_memfdmakes use offilemap_grab_foliosto grab folios in batches. This speeds up population by 8.3% due to the reduction in locking and tree walking when adding folios to the pagecache.Signed-off-by Nikita Kalyazin <[email protected]>---
virt/kvm/guest_memfd.c| 176 +++++++++++++++++++++++++++++++++--------1 file changed, 143 insertions(+), 33 deletions(-)virt/kvm/guest_memfd.c +3
-0@@ -102,17 +102,134 @@ static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index) return filemap_grab_folio(inode->i_mapping, index); } +/* +* Returns locked folios on success. The caller is responsible for +* setting the up-to-date flag before the memory is mapped into the guest.
+ * There is no backing storage for the memory, so the folios will remain
+ * up-to-date until they're removed.
+ *
+ * Ignore accessed, referenced, and dirty flags. The memory is
+ * unevictable and there is no storage to write back to.
+ */virt/kvm/guest_memfd.c +6
-0+static int kvm_gmem_get_folios (struct inode *inode,pgoff_t index, + struct folio **folios, int num) +{ + return filemap_grab_folios(inode->i_mapping, index, folios, num); +} +#if
defined(CONFIG_KVM_GENERIC_PRIVATE_MEM)&& !defined(CONFIG_KVM_AMD_SEV)virt/kvm/guest_memfd.c +32
-0+static int kvm_kmem_gmem_write_inner (struct inode *inode,pgoff_t index, + const void __user *buf, + struct folio **folios, int num) +{ + int ret, i, num_grabbed, num_written; + + num_grabbed = kvm_gmem_get_folios(inode, index, folios, num); + if (num_grabbed < 0) + return num_grabbed; + + for (i = 0; i < num_grabbed; i++) { + struct folio *folio = folios[i]; + void *vaddr; + + if (folio_test_hwpoison(folio)) { + folio_unlock(folio); + folio_put(folio); + ret = -EFAULT; + break; + } + + if (folio_test_uptodate(folio)) { + folio_unlock(folio); + folio_put(folio); + ret = -ENOSPC; + break; + } + + folio_unlock(folio); + + vaddr = kmap_local_folio(folio, 0); + ret = copy_from_user(vaddr, buf + (i << PAGE_SHIFT),
PAGE_SIZE);
+ if (ret)
+ ret =-EINVAL;
+kunmap_local(vaddr);virt/kvm/guest_memfd.c +21
-0+ + if (ret) { + folio_put(folio); + break; + } else { + kvm_gmem_mark_prepared(folio); + folio_put(folio); + } + } + + num_written = i; + + for (i = num_written; i < num_grabbed; i++) { + folio_unlock(folios[i]); + folio_put(folios[i]); + } + + return num_written ?: ret; +} + +static struct folio *kvm_kmem_gmem_write_folio(struct inode *inode,Quoted 3 lines
pgoff_tindex,
+ const char__user*buf)This could probably be rewritten as:
struct folio
*p_folio;
int ret;
Quoted 8 lines
ret =
kvm_kmem_gmem_write_inner(inode, index, buf, &p_folio, 1);if (ret == 1)
returnp_folio;
else
returnERR_PTR(ret);Would remove a few lines of duplicated code and use only one prototype.
Indeed! Thanks for the suggestion, will apply in the next revision.
Quoted 148 lines
Mike
+{
Quoted 144 lines
+ struct folio *folio;
+ void *vaddr;
+ int ret = 0;virt/kvm/guest_memfd.c +40
-0+ + folio = kvm_gmem_get_folio(inode, index); + if (IS_ERR(folio)) + return ERR_PTR(-EFAULT); + + if (folio_test_hwpoison(folio)) { + ret = -EFAULT; + goto out_unlock_put; + } + + if (folio_test_uptodate(folio)) { + ret = -ENOSPC; + goto out_unlock_put; + } + + folio_unlock(folio); + + vaddr = kmap_local_folio(folio, 0); + ret = copy_from_user(vaddr, buf, PAGE_SIZE); + if (ret) + ret = -EINVAL; + kunmap_local(vaddr); + + if (ret) { + folio_put(folio); + kvm_gmem_mark_prepared(folio); + goto out_err; + } + + folio_put(folio); + + return folio; + +out_unlock_put: + folio_unlock(folio); + folio_put(folio); +out_err: + return ERR_PTR(ret); +} +static
ssize_tkvm_kmem_gmem_write(struct file *file, const char __user *buf, size_t count, loff_t *offset)
{
+ struct inode *inode =file_inode(file);
+ int ret = 0,batch_size=FILEMAP_GET_FOLIOS_BATCH_SIZE;pgoff_tstart, end, index;
-ssize_tret = 0;if (!PAGE_ALIGNED(*offset) || !PAGE_ALIGNED(count)) return
-EINVAL;- if (*offset + count > i_size_read(file_inode(file))) + if (*offset + count >
i_size_read(inode)) return-EINVAL;if (!buf)
virt/kvm/guest_memfd.c +2
-3@@ -123,9 +240,8 @@ static ssize_t kvm_kmem_gmem_write(struct file *file, const char __user *buf, filemap_invalidate_lock(file->f_mapping); - for (index = start; index < end; ) { - struct folio *folio; - void *vaddr; + for (index = start; index + batch_size - 1 < end; ) { + struct folio *folios[FILEMAP_GET_FOLIOS_BATCH_SIZE] = {NULL }; pgoff_t buf_offset = (index - start) << PAGE_SHIFT;if (
signal_pending(current)) {virt/kvm/guest_memfd.c +17
-26@@ -133,46 +249,40 @@ static ssize_t kvm_kmem_gmem_write(struct file *file, const char __user *buf, goto out; } - folio = kvm_gmem_get_folio(file_inode(file), index); - if (IS_ERR(folio)) { - ret = -EFAULT; + ret = kvm_kmem_gmem_write_inner(inode, index, buf + buf_offset, folios, batch_size); + if (ret < 0) goto out; - } - if (folio_test_hwpoison(folio)) { - folio_unlock(folio); - folio_put(folio); - ret = -EFAULT; + index += ret; + if (ret < batch_size) + break; + } + + for (; index < end; index++) { + struct folio *folio; + pgoff_t buf_offset = (index - start) << PAGE_SHIFT; + + if (signal_pending(current)) { + ret = -EINTR; goto out; } - if (folio_test_uptodate(folio)) { - folio_unlock(folio); - folio_put(folio); - ret = -ENOSPC; + folio = kvm_kmem_gmem_write_folio(inode, index, + buf + buf_offset); + if (IS_ERR(folio)) { + ret = PTR_ERR(folio); goto out; } - - folio_unlock(folio); - - vaddr = kmap_local_folio(folio, 0); - ret = copy_from_user(vaddr, buf + buf_offset, PAGE_SIZE); - if (ret) - ret = -EINVAL; - kunmap_local(vaddr); - - kvm_gmem_mark_prepared(folio); - folio_put(folio); - - index = folio_next_index(folio); - *offset += PAGE_SIZE; }out:
filemap_invalidate_unlock(file->f_mapping); + if (index > start) {
+ *offset += (index - start) <<PAGE_SHIFT; + return (index - start) <<PAGE_SHIFT; + }- return ret && start == (*offset >>
PAGE_SHIFT) ? - ret : *offset - (start <<PAGE_SHIFT); + return ret;
}
#endif
Quoted 4 lines · Nikita Kalyazin
Based on David's suggestion for speeding up
guest_memfdmemory population [1] made at theguest_memfdupstream call on 5 Dec 2024 [2], this addsfilemap_grab_foliosthat grabs multiple folios at a time.
Hi,
Quoted 10 lines
Motivation
When profiling
guest_memfdpopulation and comparing the results with population of anonymous memory viaUFFDIO_COPY, I observed that the former was up to 20% slower, mainly due to adding newly allocated pages to the pagecache. As far as I can see, the two main contributors to it are pagecache locking and tree traversals needed for every folio. The RFC attempts to partially mitigate those by adding multiple folios at a time to the pagecache.
Quoted 6 lines
Testing
With the change applied, I was able to observe a 10.3% (708 to 635 ms) speedup in a selftest that populated 3GiB
guest_memfdand a 9.5% (990 to 904 ms) speedup when restoring a 3GiBguest_memfdVM snapshot using a custom Firecracker version, both on Intel Ice Lake.
Does that mean that it's still 10% slower (based on the 20% above), or were the 20% from a different micro-benchmark?
Quoted 8 lines
Limitations
While
filemap_grab_folioshandles THP/large folios internally and deals with reclaim artifacts in the pagecache (shadows), for simplicity reasons, the RFC does not support those as it demonstrates the optimisation applied toguest_memfd, which only uses small folios and does not support reclaim at the moment.
It might be worth pointing out that, while support for larger folios is in the works, there will be scenarios where small folios are unavoidable in the future (mixture of shared and private memory).
How hard would it be to just naturally support large folios as well?
We do have memfd_pin_folios() that can deal with that and provides a slightly similar interface (struct folio **folios).
For reference, the interface is:
long memfd_pin_folios(struct file *memfd, loff_t start, loff_t end,
struct folio **folios, unsigned int max_folios, pgoff_t *offset)
Maybe what you propose could even be used to further improve memfd_pin_folios() internally? However, it must do this FOLL_PIN thingy, so it must process each and every folio it processed.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 7 lines · David Hildenbrand
Quoted 4 lines · Nikita Kalyazin
Based on David's suggestion for speeding up
guest_memfdmemory population [1] made at theguest_memfdupstream call on 5 Dec 2024 [2], this addsfilemap_grab_foliosthat grabs multiple folios at a time.Hi,
Hi :)
Quoted 13 lines
Quoted 13 lines
Motivation
When profiling
guest_memfdpopulation and comparing the results with population of anonymous memory viaUFFDIO_COPY, I observed that the former was up to 20% slower, mainly due to adding newly allocated pages to the pagecache. As far as I can see, the two main contributors to it are pagecache locking and tree traversals needed for every folio. The RFC attempts to partially mitigate those by adding multiple folios at a time to the pagecache.Testing
Quoted 7 lines
Quoted 4 lines
With the change applied, I was able to observe a 10.3% (708 to 635 ms) speedup in a selftest that populated 3GiB
guest_memfdand a 9.5% (990 to 904 ms) speedup when restoring a 3GiBguest_memfdVM snapshot using a custom Firecracker version, both on Intel Ice Lake.Does that mean that it's still 10% slower (based on the 20% above), or were the 20% from a different micro-benchmark?
Yes, it is still slower:
- isolated/selftest: 2.3%
- Firecracker setup: 8.9%
Not sure why the values are so different though. I'll try to find an explanation.
Quoted 9 lines
Quoted 8 lines
Limitations
While
filemap_grab_folioshandles THP/large folios internally and deals with reclaim artifacts in the pagecache (shadows), for simplicity reasons, the RFC does not support those as it demonstrates the optimisation applied toguest_memfd, which only uses small folios and does not support reclaim at the moment.
Quoted 5 lines
It might be worth pointing out that, while support for larger folios is in the works, there will be scenarios where small folios are unavoidable in the future (mixture of shared and private memory).
How hard would it be to just naturally support large folios as well?
I don't think it's going to be impossible. It's just one more dimension that needs to be handled. __filemap_add_folio logic is already rather complex, and processing multiple folios while also splitting when necessary correctly looks substantially convoluted to me. So my idea was to discuss/validate the multi-folio approach first before rolling the sleeves up.
Quoted 5 lines
We do have
memfd_pin_folios()that can deal with that and provides a slightly similar interface (struct folio **folios).For reference, the interface is:
Quoted 7 lines
long memfd_pin_folios(struct file *memfd,
loff_tstart,loff_tend,
struct folio **folios, unsigned intmax_folios,pgoff_t*offset)Maybe what you propose could even be used to further improve
memfd_pin_folios()internally? However, it must do thisFOLL_PINthingy, so it must process each and every folio it processed.
Thanks for the pointer. Yeah, I see what you mean. I guess, it can potentially allocate/add folios in a batch and then pin them? Although swap/readahead logic may make it more difficult to implement.
Quoted 4 lines
Signature · 3 lines
Cheers, David / dhildenb
Quoted 32 lines · Nikita Kalyazin
Quoted 7 lines · David Hildenbrand
Quoted 4 lines · Nikita Kalyazin
Based on David's suggestion for speeding up
guest_memfdmemory population [1] made at theguest_memfdupstream call on 5 Dec 2024 [2], this addsfilemap_grab_foliosthat grabs multiple folios at a time.Hi,
Hi :)
Quoted 13 lines
Quoted 13 lines
Motivation
When profiling
guest_memfdpopulation and comparing the results with population of anonymous memory viaUFFDIO_COPY, I observed that the former was up to 20% slower, mainly due to adding newly allocated pages to the pagecache. As far as I can see, the two main contributors to it are pagecache locking and tree traversals needed for every folio. The RFC attempts to partially mitigate those by adding multiple folios at a time to the pagecache.Testing
Quoted 7 lines
Quoted 4 lines
With the change applied, I was able to observe a 10.3% (708 to 635 ms) speedup in a selftest that populated 3GiB
guest_memfdand a 9.5% (990 to 904 ms) speedup when restoring a 3GiBguest_memfdVM snapshot using a custom Firecracker version, both on Intel Ice Lake.Does that mean that it's still 10% slower (based on the 20% above), or were the 20% from a different micro-benchmark?
Quoted 6 lines
Yes, it is still slower:
- isolated/selftest: 2.3%
- Firecracker setup: 8.9%Not sure why the values are so different though. I'll try to find an explanation.
The 2.3% looks very promising.
Quoted 14 lines
Quoted 14 lines
Quoted 9 lines
Limitations
While
filemap_grab_folioshandles THP/large folios internally and deals with reclaim artifacts in the pagecache (shadows), for simplicity reasons, the RFC does not support those as it demonstrates the optimisation applied toguest_memfd, which only uses small folios and does not support reclaim at the moment.It might be worth pointing out that, while support for larger folios is in the works, there will be scenarios where small folios are unavoidable in the future (mixture of shared and private memory).
Quoted 8 lines
Quoted 1 line
How hard would it be to just naturally support large folios as well?
I don't think it's going to be impossible. It's just one more dimension that needs to be handled.
__filemap_add_foliologic is already rather complex, and processing multiple folios while also splitting when necessary correctly looks substantially convoluted to me. So my idea was to discuss/validate the multi-folio approach first before rolling the sleeves up.
We should likely try making this as generic as possible, meaning we'll support roughly what filemap_grab_folio() would have supported (e.g., also large folios).
Now I find filemap_get_folios_contig() [thas is already used in memfd code], and wonder if that could be reused/extended fairly easily.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 33 lines · David Hildenbrand
Quoted 32 lines · Nikita Kalyazin
Quoted 7 lines · David Hildenbrand
Quoted 4 lines · Nikita Kalyazin
Based on David's suggestion for speeding up
guest_memfdmemory population [1] made at theguest_memfdupstream call on 5 Dec 2024 [2], this addsfilemap_grab_foliosthat grabs multiple folios at a time.Hi,
Hi :)
Quoted 20 lines
Quoted 11 lines
Motivation
When profiling
guest_memfdpopulation and comparing the results with population of anonymous memory viaUFFDIO_COPY, I observed that the former was up to 20% slower, mainly due to adding newly allocated pages to the pagecache. As far as I can see, the two main contributors to it are pagecache locking and tree traversals needed for every folio. The RFC attempts to partially mitigate those by adding multiple folios at a time to the pagecache.Quoted 6 lines
Testing
With the change applied, I was able to observe a 10.3% (708 to 635 ms) speedup in a selftest that populated 3GiB
guest_memfdand a 9.5% (990 to 904 ms) speedup when restoring a 3GiBguest_memfdVM snapshot using a custom Firecracker version, both on Intel Ice Lake.Does that mean that it's still 10% slower (based on the 20% above), or were the 20% from a different micro-benchmark?
Quoted 8 lines
Quoted 6 lines
Yes, it is still slower:
- isolated/selftest: 2.3%
- Firecracker setup: 8.9%Not sure why the values are so different though. I'll try to find an explanation.
The 2.3% looks very promising.
It does. I sorted out my Firecracker setup and saw a similar figure there, which made me more confident.
Quoted 23 lines
Quoted 14 lines
Quoted 14 lines
Quoted 9 lines
Limitations
While
filemap_grab_folioshandles THP/large folios internally and deals with reclaim artifacts in the pagecache (shadows), for simplicity reasons, the RFC does not support those as it demonstrates the optimisation applied toguest_memfd, which only uses small folios and does not support reclaim at the moment.It might be worth pointing out that, while support for larger folios is in the works, there will be scenarios where small folios are unavoidable in the future (mixture of shared and private memory).
Quoted 8 lines
Quoted 1 line
How hard would it be to just naturally support large folios as well?
I don't think it's going to be impossible. It's just one more dimension that needs to be handled.
__filemap_add_foliologic is already rather complex, and processing multiple folios while also splitting when necessary correctly looks substantially convoluted to me. So my idea was to discuss/validate the multi-folio approach first before rolling the sleeves up.
Quoted 5 lines
We should likely try making this as generic as possible, meaning we'll support roughly what
filemap_grab_folio()would have supported (e.g., also large folios).Now I find
filemap_get_folios_contig()[thas is already used in memfd code], and wonder if that could be reused/extended fairly easily.
Fair, I will see into how it could be made generic.
Quoted 5 lines
Signature · 3 lines
Cheers, David / dhildenb