mm: memfd_luo: fixes for folio flag preservation
Subsystems (3)
Pratyush Yadav
maintainercover mm: memfd_luo: fixes for folio flag preservation
From: "Pratyush Yadav (Google)" <[email protected]>
Hi,
This series contains a couple fixes for flag preservation for memfd live update.
The first patch fixes memfd preservation when fallocate() was used to pre-allocate some pages. For these memfds, all the writes to fallocated pages touched after preserve were lost.
The second patch fixes dirty flag tracking. If the dirty flag is not tracked correctly, the next kernel might incorrectly reclaim some folios under memory pressure, losing user data. This is a theoretical bug that I observed when reading the code, and haven't been able to reproduce it.
Regards,
Pratyush Yadav
Pratyush Yadav (Google) (2):
mm: memfd_luo: always make all folios uptodate
mm: memfd_luo: always dirty all folios
mm/memfd_luo.c | 49 +++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 43 insertions(+), 6 deletions(-)
base-commit: 8bf22c33e7a172fbc72464f4cc484d23a6b412ba
Signature · 1 line
2.53.0.371.g1d285c8824-goog
Pratyush Yadav
maintainerPATCH 1/2 mm: memfd_luo: always make all folios uptodate
From: "Pratyush Yadav (Google)" <[email protected]>
When a folio is added to a shmem file via fallocate, it is not zeroed on allocation. This is done as a performance optimization since it is possible the folio will never end up being used at all. When the folio is used, shmem checks for the uptodate flag, and if absent, zeroes the folio (and sets the flag) before returning to user.
With LUO, the flags of each folio are saved at preserve time. It is possible to have a memfd with some folios fallocated but not uptodate. For those, the uptodate flag doesn't get saved. The folios might later end up being used and become uptodate. They would get passed to the next kernel via KHO correctly since they did get preserved. But they won't have the MEMFD_LUO_FOLIO_UPTODATE flag.
This means that when the memfd is retrieved, the folios will be added to the shmem file without the uptodate flag. They will be zeroed before first use, losing the data in those folios.
Since we take a big performance hit in allocating, zeroing, and pinning all folios at prepare time anyway, take some more and zero all non-uptodate ones too.
Later when there is a stronger need to make prepare faster, this can be optimized.
To avoid racing with another uptodate operation, take the folio lock.
--- mm/memfd_luo.c | 25 +++++++++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-)
mm/memfd_luo.c +23 -2
@@ -152,10 +152,31 @@ static int memfd_luo_preserve_folios(struct file *file,if (err) goto err_unpreserve; +folio_lock (folio); +if (folio_test_dirty (folio)) flags |=MEMFD_LUO_FOLIO_DIRTY ; -if (folio_test_uptodate (folio)) - flags |=MEMFD_LUO_FOLIO_UPTODATE ; + +/* +* If the folio is not uptodate, it was fallocated but never +* used. Saving this flag at prepare() doesn't work since it +* might change later when someone uses the folio. +* +* Since we have taken the performance penalty of allocating, +* zeroing, and pinning all the folios in the holes, take a bit +* more and zero all non-uptodate folios too. +* +* NOTE: For someone looking to improve preserve performance, +* this is a good place to look. +*/ +if (!folio_test_uptodate (folio)) { +folio_zero_range (folio,0 ,folio_size (folio)); +flush_dcache_folio (folio); +folio_mark_uptodate (folio); + } + flags |=MEMFD_LUO_FOLIO_UPTODATE ; + +folio_unlock (folio); pfolio->pfn =folio_pfn (folio); pfolio->flags = flags;
Signature · 1 line
2.53.0.371.g1d285c8824-goog
Re: [PATCH 1/2] mm: memfd_luo: always make all folios uptodate
Quoted 23 lines · Pratyush Yadav
From: "Pratyush Yadav (Google)" <[email protected]>
When a folio is added to a shmem file via fallocate, it is not zeroed on allocation. This is done as a performance optimization since it is possible the folio will never end up being used at all. When the folio is used, shmem checks for the uptodate flag, and if absent, zeroes the folio (and sets the flag) before returning to user.
With LUO, the flags of each folio are saved at preserve time. It is possible to have a memfd with some folios fallocated but not uptodate. For those, the uptodate flag doesn't get saved. The folios might later end up being used and become uptodate. They would get passed to the next kernel via KHO correctly since they did get preserved. But they won't have the
MEMFD_LUO_FOLIO_UPTODATEflag.This means that when the memfd is retrieved, the folios will be added to the shmem file without the uptodate flag. They will be zeroed before first use, losing the data in those folios.
Since we take a big performance hit in allocating, zeroing, and pinning all folios at prepare time anyway, take some more and zero all non-uptodate ones too.
Quoted 8 lines
Later when there is a stronger need to make prepare faster, this can be optimized.
To avoid racing with another uptodate operation, take the folio lock.
Fixes b3749f174d68 ("mm: memfd_luo: allow preserving memfd")Signed-off-by Pratyush Yadav (Google) <[email protected]>
Quoted 20 lines
--- mm/memfd_luo.c | 25 +++++++++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-)mm/memfd_luo.c +4
-2@@ -152,10 +152,31 @@ static int memfd_luo_preserve_folios(struct file *file,if (err) goto err_unpreserve; +folio_lock (folio); +if (folio_test_dirty (folio)) flags |=MEMFD_LUO_FOLIO_DIRTY ; -if (folio_test_uptodate (folio)) - flags |=MEMFD_LUO_FOLIO_UPTODATE ; + +/*
mm/memfd_luo.c static int memfd_luo_preserve_folios(struct file *file, · line 170
Quoted 10 lines
mm/memfd_luo.c +10
-0+* If the folio is not uptodate, it was fallocated but never +* used. Saving this flag at prepare() doesn't work since it +* might change later when someone uses the folio. +* +* Since we have taken the performance penalty of allocating, +* zeroing, and pinning all the folios in the holes, take a bit +* more and zero all non-uptodate folios too. +* +* NOTE: For someone looking to improve preserve performance, +* this is a good place to look.
I'd add a larger comment above memfd_luo_preserve_folios() that says that it allocates, pins etc and fold the last two paragraphs of this comment there.
Quoted 15 lines
mm/memfd_luo.c +9
-0+*/ +if (!folio_test_uptodate (folio)) { +folio_zero_range (folio,0 ,folio_size (folio)); +flush_dcache_folio (folio); +folio_mark_uptodate (folio); + } + flags |=MEMFD_LUO_FOLIO_UPTODATE ; + +folio_unlock (folio); pfolio->pfn =folio_pfn (folio); pfolio->flags = flags;Signature · 1 line
2.53.0.371.g1d285c8824-goog
Signature · 2 lines
Sincerely yours, Mike.
Pratyush Yadav
maintainerRe: [PATCH 1/2] mm: memfd_luo: always make all folios uptodate
Quoted 1 line · Mike Rapoport
On Mon, Feb 23, 2026 at 06:39:28PM +0100, Pratyush Yadav wrote:
[...]
Quoted 8 lines
Quoted 8 lines
mm/memfd_luo.c +8
-0+ +/* +* If the folio is not uptodate, it was fallocated but never +* used. Saving this flag at prepare() doesn't work since it +* might change later when someone uses the folio. +* +* Since we have taken the performance penalty of allocating, +* zeroing, and pinning all the folios in the holes, take a bit
Quoted 8 lines
mm/memfd_luo.c static int memfd_luo_preserve_folios(struct file *file, · line 170
Quoted 4 lines
mm/memfd_luo.c +4
-0+* more and zero all non-uptodate folios too. +* +* NOTE: For someone looking to improve preserve performance, +* this is a good place to look. I'd add a larger comment above
memfd_luo_preserve_folios()that says that it allocates, pins etc and fold the last two paragraphs of this comment there.
How about this:
/*
* If the folio is not uptodate, it was fallocated but never
* used. Saving this flag at prepare() doesn't work since it
* might change later when the folio is used. Make it uptodate
* now to avoid this problem.
*/
if (!folio_test_uptodate(folio)) {
And the comment above memfd_pin_folios() gets this:
* NOTE: For someone looking to improve preserve performance, this is a
* good place to look. Also look at the folio zeroing below.
[...]
Signature · 2 lines
Regards, Pratyush Yadav
Mike Rapoport
maintainerRe: [PATCH 1/2] mm: memfd_luo: always make all folios uptodate
Quoted 31 lines · Pratyush Yadav
Quoted 1 line · Mike Rapoport
On Mon, Feb 23, 2026 at 06:39:28PM +0100, Pratyush Yadav wrote:
[...]
Quoted 8 lines
Quoted 8 lines
mm/memfd_luo.c +8
-0+ +/* +* If the folio is not uptodate, it was fallocated but never +* used. Saving this flag at prepare() doesn't work since it +* might change later when someone uses the folio. +* +* Since we have taken the performance penalty of allocating, +* zeroing, and pinning all the folios in the holes, take a bit Quoted 8 lines
mm/memfd_luo.c static int memfd_luo_preserve_folios(struct file *file, · line 170
Quoted 4 lines
mm/memfd_luo.c +4
-0+* more and zero all non-uptodate folios too. +* +* NOTE: For someone looking to improve preserve performance, +* this is a good place to look. I'd add a larger comment above
memfd_luo_preserve_folios()that says that it allocates, pins etc and fold the last two paragraphs of this comment there.How about this:
/*
* If the folio is not uptodate, it was fallocated but never
* used. Saving this flag atprepare()doesn't work since it
* might change later when the folio is used. Make it uptodate
* now to avoid this problem.
*/
if (!folio_test_uptodate(folio)) {
Quoted 4 lines
And the comment above
memfd_pin_folios()gets this:* NOTE: For someone looking to improve preserve performance, this is a
* good place to look. Also look at the folio zeroing below.
Looks great to me
Quoted 5 lines
[...]
Signature · 2 lines
Regards, Pratyush Yadav
Signature · 2 lines
Sincerely yours, Mike.
Pratyush Yadav
maintainerPATCH 2/2 mm: memfd_luo: always dirty all folios
In reply to Pratyush YadavFrom: "Pratyush Yadav (Google)" <[email protected]>
A dirty folio is one which has been written to. A clean folio is its opposite. Since a clean folio has no user data, it can be freed under memory pressure.
memfd preservation with LUO saves the flag at preserve(). This is problematic. The folio might get dirtied later. Saving it at freeze() also doesn't work, since the dirty bit from PTE is normally synced at unmap and there might still be mappings of the file at freeze().
To see why this is a problem, say a folio is clean at preserve, but gets dirtied later. The serialized state of the folio will mark it as clean. After retrieve, the next kernel will see the folio as clean and might try to reclaim it under memory pressure. This will result in losing user data.
Mark all folios of the file as dirty, and always set the MEMFD_LUO_FOLIO_DIRTY flag. This comes with the side effect of making all clean folios un-reclaimable. This is a cost that has to be paid for participants of live update. It is not expected to be a common use case to preserve a lot of clean folios anyway.
Since the value of pfolio->flags is a constant now, drop the flags variable and set it directly.
--- mm/memfd_luo.c | 26 +++++++++++++++++++++----- 1 file changed, 21 insertions(+), 5 deletions(-)
mm/memfd_luo.c +21 -5
@@ -146,7 +146,6 @@ static int memfd_luo_preserve_folios(struct file *file,for (i =0 ; i < nr_folios; i++) {struct memfd_luo_folio_ser *pfolio = &folios_ser[i];struct folio *folio = folios[i]; -unsigned int flags =0 ; err =kho_preserve_folio (folio);if (err)
@@ -154,8 +153,26 @@ static int memfd_luo_preserve_folios(struct file *file,folio_lock (folio); -if (folio_test_dirty (folio)) - flags |=MEMFD_LUO_FOLIO_DIRTY ; +/* +* A dirty folio is one which has been written to. A clean folio +* is its opposite. Since a clean folio does not carry user +* data, it can be freed by page reclaim under memory pressure. +* +* Saving the dirty flag at prepare() time doesn't work since it +* can change later. Saving it at freeze() also won't work +* because the dirty bit is normally synced at unmap and there +* might still be a mapping of the file at freeze(). +* +* To see why this is a problem, say a folio is clean at +* preserve, but gets dirtied later. The pfolio flags will mark +* it as clean. After retrieve, the next kernel might try to +* reclaim this folio under memory pressure, losing user data. +* +* Unconditionally mark it dirty to avoid this problem. This +* comes at the cost of making clean folios un-reclaimable after +* live update. +*/ +folio_mark_dirty (folio);/* * If the folio is not uptodate, it was fallocated but never
@@ -174,12 +191,11 @@ static int memfd_luo_preserve_folios(struct file *file,flush_dcache_folio (folio);folio_mark_uptodate (folio); } - flags |=MEMFD_LUO_FOLIO_UPTODATE ;folio_unlock (folio); pfolio->pfn =folio_pfn (folio); - pfolio->flags = flags; + pfolio->flags =MEMFD_LUO_FOLIO_DIRTY |MEMFD_LUO_FOLIO_UPTODATE ; pfolio->index = folio->index; }
Signature · 1 line
2.53.0.371.g1d285c8824-goog
Re: [PATCH 2/2] mm: memfd_luo: always dirty all folios
Quoted 23 lines · Pratyush Yadav
From: "Pratyush Yadav (Google)" <[email protected]>
A dirty folio is one which has been written to. A clean folio is its opposite. Since a clean folio has no user data, it can be freed under memory pressure.
memfd preservation with LUO saves the flag at
preserve(). This is problematic. The folio might get dirtied later. Saving it atfreeze()also doesn't work, since the dirty bit from PTE is normally synced at unmap and there might still be mappings of the file atfreeze().To see why this is a problem, say a folio is clean at preserve, but gets dirtied later. The serialized state of the folio will mark it as clean. After retrieve, the next kernel will see the folio as clean and might try to reclaim it under memory pressure. This will result in losing user data.
Mark all folios of the file as dirty, and always set the
MEMFD_LUO_FOLIO_DIRTYflag. This comes with the side effect of making all clean folios un-reclaimable. This is a cost that has to be paid for participants of live update. It is not expected to be a common use case to preserve a lot of clean folios anyway.
Quoted 6 lines
Since the value of
pfolio->flagsis a constant now, drop the flags variable and set it directly.Fixes b3749f174d68 ("mm: memfd_luo: allow preserving memfd")Signed-off-by Pratyush Yadav (Google) <[email protected]>
Quoted 31 lines
--- mm/memfd_luo.c | 26 +++++++++++++++++++++----- 1 file changed, 21 insertions(+), 5 deletions(-)mm/memfd_luo.c +9
-3@@ -146,7 +146,6 @@ static int memfd_luo_preserve_folios(struct file *file,for (i =0 ; i < nr_folios; i++) {struct memfd_luo_folio_ser *pfolio = &folios_ser[i];struct folio *folio = folios[i]; -unsigned int flags =0 ; err =kho_preserve_folio (folio);if (err)@@ -154,8 +153,26 @@ static int memfd_luo_preserve_folios(struct file *file,folio_lock (folio); -if (folio_test_dirty (folio)) - flags |=MEMFD_LUO_FOLIO_DIRTY ; +/* +* A dirty folio is one which has been written to. A clean folio +* is its opposite. Since a clean folio does not carry user +* data, it can be freed by page reclaim under memory pressure. +* +* Saving the dirty flag at prepare() time doesn't work since it +* can change later. Saving it at freeze() also won't work +* because the dirty bit is normally synced at unmap and there +* might still be a mapping of the file at freeze().
mm/memfd_luo.c static int memfd_luo_preserve_folios(struct file *file, · line 174
Quoted 10 lines
mm/memfd_luo.c +10
-0+* +* To see why this is a problem, say a folio is clean at +* preserve, but gets dirtied later. The pfolio flags will mark +* it as clean. After retrieve, the next kernel might try to +* reclaim this folio under memory pressure, losing user data. +* +* Unconditionally mark it dirty to avoid this problem. This +* comes at the cost of making clean folios un-reclaimable after +* live update. +*/
Can we make the comment here shorter to only contain the gist of the issue?
Quoted 21 lines
mm/memfd_luo.c +2
-2+folio_mark_dirty (folio);/* * If the folio is not uptodate, it was fallocated but never @@ -174,12 +191,11 @@ static int memfd_luo_preserve_folios(struct file *file,flush_dcache_folio (folio);folio_mark_uptodate (folio); } - flags |=MEMFD_LUO_FOLIO_UPTODATE ;folio_unlock (folio); pfolio->pfn =folio_pfn (folio); - pfolio->flags = flags; + pfolio->flags =MEMFD_LUO_FOLIO_DIRTY |MEMFD_LUO_FOLIO_UPTODATE ; pfolio->index = folio->index; }Signature · 1 line
2.53.0.371.g1d285c8824-goog
Signature · 2 lines
Sincerely yours, Mike.
Pratyush Yadav
maintainerRe: [PATCH 2/2] mm: memfd_luo: always dirty all folios
Hi Mike,
Quoted 1 line · Mike Rapoport
On Mon, Feb 23, 2026 at 06:39:29PM +0100, Pratyush Yadav wrote:
[...]
Quoted 15 lines
Quoted 15 lines
mm/memfd_luo.c +13
-2-if (folio_test_dirty (folio)) - flags |=MEMFD_LUO_FOLIO_DIRTY ; +/* +* A dirty folio is one which has been written to. A clean folio +* is its opposite. Since a clean folio does not carry user +* data, it can be freed by page reclaim under memory pressure. +* +* Saving the dirty flag at prepare() time doesn't work since it +* can change later. Saving it at freeze() also won't work +* because the dirty bit is normally synced at unmap and there +* might still be a mapping of the file at freeze(). +* +* To see why this is a problem, say a folio is clean at +* preserve, but gets dirtied later. The pfolio flags will mark +* it as clean. After retrieve, the next kernel might try to
Quoted 8 lines
mm/memfd_luo.c static int memfd_luo_preserve_folios(struct file *file, · line 174
Quoted 6 lines
mm/memfd_luo.c +6
-0+* reclaim this folio under memory pressure, losing user data. +* +* Unconditionally mark it dirty to avoid this problem. This +* comes at the cost of making clean folios un-reclaimable after +* live update. +*/ Can we make the comment here shorter to only contain the gist of the issue?
Is this any better? Or should I try to make it shorter still?
/*
* Tracking the dirty flag of the folio is difficult since it is
* normally synced at unmap and there might still be mappings of
* the file alive.
*
* Not tracking it correctly can cause a dirty folio to be
* restored as clean after KHO. The next kernel might then try
* to reclaim it, losing user data.
*
* Unconditionally mark the folio dirty to avoid this. This
* comes at the cost of making clean folios un-reclaimable. */
[...]
Signature · 2 lines
Regards, Pratyush Yadav
Mike Rapoport
maintainerRe: [PATCH 2/2] mm: memfd_luo: always dirty all folios
Quoted 30 lines · Pratyush Yadav
Hi Mike,
Quoted 1 line · Mike Rapoport
On Mon, Feb 23, 2026 at 06:39:29PM +0100, Pratyush Yadav wrote:
[...]
Quoted 15 lines
Quoted 15 lines
mm/memfd_luo.c +13
-2-if (folio_test_dirty (folio)) - flags |=MEMFD_LUO_FOLIO_DIRTY ; +/* +* A dirty folio is one which has been written to. A clean folio +* is its opposite. Since a clean folio does not carry user +* data, it can be freed by page reclaim under memory pressure. +* +* Saving the dirty flag at prepare() time doesn't work since it +* can change later. Saving it at freeze() also won't work +* because the dirty bit is normally synced at unmap and there +* might still be a mapping of the file at freeze(). +* +* To see why this is a problem, say a folio is clean at +* preserve, but gets dirtied later. The pfolio flags will mark +* it as clean. After retrieve, the next kernel might try to Quoted 8 lines
mm/memfd_luo.c static int memfd_luo_preserve_folios(struct file *file, · line 174
Quoted 6 lines
mm/memfd_luo.c +6
-0+* reclaim this folio under memory pressure, losing user data. +* +* Unconditionally mark it dirty to avoid this problem. This +* comes at the cost of making clean folios un-reclaimable after +* live update. +*/ Can we make the comment here shorter to only contain the gist of the issue?
Quoted 1 line
Is this any better? Or should I try to make it shorter still?
Yep, thanks! :)
I don't think it can be made shorter without loosing information. What we might do is to add a larger comment in what state we preserve folios on top of memfd_luo_preserve_folios() and leave the code inside the function alone. Can't say I have strong feelings about it.
Quoted 18 lines
/*
* Tracking the dirty flag of the folio is difficult since it is
* normally synced at unmap and there might still be mappings of
* the file alive.
*
* Not tracking it correctly can cause a dirty folio to be
* restored as clean after KHO. The next kernel might then try
* to reclaim it, losing user data.
*
* Unconditionally mark the folio dirty to avoid this. This
* comes at the cost of making clean folios un-reclaimable. */[...]
Signature · 2 lines
Regards, Pratyush Yadav
Signature · 2 lines
Sincerely yours, Mike.