mm: add support for dropping LRU recency on process exit
Subsystems (3)
RFC PATCH v2 1/1 mm: add support for dropping LRU recency on process exit
From: Barry Song <[email protected]>
Currently, both zap_pmd and zap_pte always promote young file folios, regardless of whether the processes are dying.
However, in systems where the process recency fades upon dying, we may want to reverse this behavior. The goal is to reclaim the folios from the dying process as quickly as possible, allowing new processes to acquire memory ASAP.
For example, while Firefox is killed and LibreOffice is launched, activating Firefox's young file-backed folios makes it harder to reclaim memory that LibreOffice doesn't use at all.
On systems like Android, processes are either explicitly stopped by user action or reaped due to OOM after being inactive for a long time. These processes are unlikely to restart in the near future. Rather than promoting their folios, we skip promoting and demote their exclusive folios so that memory can be reclaimed and made available for new user-facing processes.
Users possibly do not care about the recency of a dying process. However, we still need an explicit user indication to take this action. Thus, we introduced a prctl to provide that necessary user-level hint as suggested by Johannes and David.
We observed noticeable improvements in refaults, swap-ins, and swap-outs on a hooked Android kernel. More data for this specific version will follow.
---
-v2:
* add prctl as suggested by Johannes and David
* demote exclusive file folios if drop_recency can apply -v1:
https://lore.kernel.org/linux-mm/[email protected]/
include/linux/mm_types.h | 1 + include/uapi/linux/prctl.h | 3 +++ kernel/sys.c | 16 ++++++++++++++++ mm/huge_memory.c | 12 ++++++++++-- mm/internal.h | 14 ++++++++++++++ mm/memory.c | 12 +++++++++++- 6 files changed, 55 insertions(+), 3 deletions(-)
include/linux/mm_types.h +1 -0
@@ -1733,6 +1733,7 @@ enum {* on NFS restore */ //#define MMF_EXE_FILE_CHANGED 18 /* see prctl_set_mm_exe_file() */ +#define MMF_FADE_ON_DEATH 18/* Recency is discarded on process exit */ #define MMF_HAS_UPROBES 19/* has uprobes */ #define MMF_RECALC_UPROBES 20/* MMF_HAS_UPROBES can be wrong */
include/uapi/linux/prctl.h +3 -0
@@ -364,4 +364,7 @@ struct prctl_mm_map {# define PR_TIMER_CREATE_RESTORE_IDS_ON 1# define PR_TIMER_CREATE_RESTORE_IDS_GET 2 +#define PR_SET_FADE_ON_DEATH 78 +#define PR_GET_FADE_ON_DEATH 79 +#endif /* _LINUX_PRCTL_H */
kernel/sys.c +16 -0
@@ -2658,6 +2658,22 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3,clear_bit (MMF_DISABLE_THP , &me->mm->flags);mmap_write_unlock (me->mm);break ; +case PR_GET_FADE_ON_DEATH : +if (arg2 || arg3 || arg4 || arg5) +return -EINVAL ; + error = !!test_bit (MMF_FADE_ON_DEATH , &me->mm->flags); +break ; +case PR_SET_FADE_ON_DEATH : +if (arg3 || arg4 || arg5) +return -EINVAL ; +if (mmap_write_lock_killable (me->mm)) +return -EINTR ; +if (arg2) +set_bit (MMF_FADE_ON_DEATH , &me->mm->flags); +else +clear_bit (MMF_FADE_ON_DEATH , &me->mm->flags); +mmap_write_unlock (me->mm); +break ;case PR_MPX_ENABLE_MANAGEMENT :case PR_MPX_DISABLE_MANAGEMENT :/* No longer implemented: */
mm/huge_memory.c +10 -2
@@ -2204,6 +2204,7 @@ static inline void zap_deposited_table(struct mm_struct *mm, pmd_t *pmd)int zap_huge_pmd (struct mmu_gather *tlb,struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr) { +bool drop_recency = false;pmd_t orig_pmd;spinlock_t *ptl;
@@ -2260,13 +2261,20 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,add_mm_counter (tlb->mm,mm_counter_file (folio), -HPAGE_PMD_NR ); + drop_recency =zap_need_to_drop_recency (tlb->mm);/* * Use flush_needed to indicate whether the PMD entry * is present, instead of checking pmd_present() again. */ -if (flush_needed &&pmd_young (orig_pmd) && -likely (vma_has_recency (vma))) +if (flush_needed &&pmd_young (orig_pmd) && !drop_recency && +likely (vma_has_recency (vma)))folio_mark_accessed (folio); +/* +* Userspace explicitly marks recency to fade when the process +* dies; demote exclusive file folios to aid reclamation. +*/ +if (drop_recency && !folio_maybe_mapped_shared (folio)) +deactivate_file_folio (folio); }spin_unlock (ptl);
mm/internal.h +14 -0
@@ -11,6 +11,7 @@#include <linux/khugepaged.h> #include <linux/mm.h> #include <linux/mm_inline.h> +#include <linux/oom.h> #include <linux/pagemap.h> #include <linux/pagewalk.h> #include <linux/rmap.h>
@@ -130,6 +131,19 @@ static inline int folio_nr_pages_mapped(const struct folio *folio)return atomic_read (&folio->_nr_pages_mapped) &FOLIO_PAGES_MAPPED ; } +/* +* Returns true if the process attached to the mm is dying or undergoing +* OOM reaping, and its recency—explicitly marked by userspace—will also +* fade; otherwise, returns false. +*/ +static inline bool zap_need_to_drop_recency (struct mm_struct *mm) +{ +if (!atomic_read (&mm->mm_users) ||check_stable_address_space (mm)) +return !!test_bit (MMF_FADE_ON_DEATH , &mm->flags); + +return false; +} +/* * Retrieve the first entry of a folio based on a provided entry within the * folio. We cannot rely on folio->swap as there is no guarantee that it has
mm/memory.c +11 -1
@@ -1505,6 +1505,7 @@ static __always_inline void zap_present_folio_ptes(struct mmu_gather *tlb, bool *force_flush, bool *force_break, bool *any_skipped) { struct mm_struct *mm = tlb->mm; +bool drop_recency = false;bool delay_rmap = false;if (!folio_test_anon (folio)) {
@@ -1516,9 +1517,18 @@ static __always_inline void zap_present_folio_ptes(struct mmu_gather *tlb, *force_flush = true; } } -if (pte_young (ptent) &&likely (vma_has_recency (vma))) + + drop_recency =zap_need_to_drop_recency (mm); +if (pte_young (ptent) && !drop_recency && +likely (vma_has_recency (vma)))folio_mark_accessed (folio); rss[mm_counter (folio)] -= nr; +/* +* Userspace explicitly marks recency to fade when the process dies; +* demote exclusive file folios to aid reclamation. +*/ +if (drop_recency && !folio_maybe_mapped_shared (folio)) +deactivate_file_folio (folio); }else {/* We don't need up-to-date accessed/dirty bits. */ clear_full_ptes (mm, addr, pte, nr, tlb->fullmm);
Signature · 1 line
2.39.3 (Apple Git-146)
Quoted 19 lines · Barry Song
From: Barry Song <[email protected]>
Currently, both
zap_pmdandzap_ptealways promote young file folios, regardless of whether the processes are dying.
However, in systems where the process recency fades upon dying, we may want to reverse this behavior. The goal is to reclaim the folios from the dying process as quickly as possible, allowing new processes to acquire memory ASAP.
For example, while Firefox is killed and LibreOffice is launched, activating Firefox's young file-backed folios makes it harder to reclaim memory that LibreOffice doesn't use at all.On systems like Android, processes are either explicitly stopped by user action or reaped due to OOM after being inactive for a long time. These processes are unlikely to restart in the near future. Rather than promoting their folios, we skip promoting and demote their exclusive folios so that memory can be reclaimed and made available for new user-facing processes.
Quoted 2 lines
Users possibly do not care about the recency of a dying process. However, we still need an explicit user indication to take this action.
Can you add why? It'd be nice to capture the reasons pointed out in v1 discussion as they seem important to why this isn't set as a default for all tasks.
Quoted 2 lines
Thus, we introduced a prctl to provide that necessary user-level hint as suggested by Johannes and David.
I'm not sure it really makes much of a difference if we update the lru or not in this case. Johannes point about this small change having unknown results for the larger community is certainly the best argument as to why we need this to be opt-in.
We should probably document it so that people can opt-in though :)
Quoted 4 lines
We observed noticeable improvements in refaults, swap-ins, and swap-outs on a hooked Android kernel. More data for this specific version will follow.
Looking forward to the results. What happens when I kill my app and reopen it? (close all apps, open the one that was being annoying?)
Quoted 35 lines
Cc Baolin Wang <[email protected]>Cc David Hildenbrand <[email protected]>Cc Johannes Weiner <[email protected]>Cc Matthew Wilcox (Oracle) <[email protected]>Cc Oscar Salvador <[email protected]>Cc Ryan Roberts <[email protected]>Cc Zi Yan <[email protected]>Cc Lorenzo Stoakes <[email protected]>Cc "Liam R. Howlett" <[email protected]>Cc Vlastimil Babka <[email protected]>Cc Mike Rapoport <[email protected]>Cc Suren Baghdasaryan <[email protected]>Cc Michal Hocko <[email protected]>Signed-off-by Barry Song <[email protected]>---
-v2:
* add prctl as suggested by Johannes and David
* demote exclusive file folios ifdrop_recencycan apply -v1:
https://lore.kernel.org/linux-mm/[email protected]/include/linux/mm_types.h | 1 + include/uapi/linux/prctl.h | 3 +++ kernel/sys.c | 16 ++++++++++++++++ mm/huge_memory.c | 12 ++++++++++-- mm/internal.h | 14 ++++++++++++++ mm/memory.c | 12 +++++++++++- 6 files changed, 55 insertions(+), 3 deletions(-)include/linux/mm_types.h +0
-0
include/linux/mm_types.h enum { · line 1736
Quoted 4 lines
include/linux/mm_types.h +1
-0@@ -1733,6 +1733,7 @@ enum {* on NFS restore */ //#define MMF_EXE_FILE_CHANGED 18 /* see prctl_set_mm_exe_file() */ +#define MMF_FADE_ON_DEATH 18/* Recency is discarded on process exit */
Why is recency not in the MMF name? Why not MMF_NO_RECENCY or something?
I guess we are back to no space in this flag.
Quoted 20 lines
include/linux/mm_types.h +0
-0#define MMF_HAS_UPROBES 19/* has uprobes */ #define MMF_RECALC_UPROBES 20/* MMF_HAS_UPROBES can be wrong */ include/uapi/linux/prctl.h +3
-0@@ -364,4 +364,7 @@ struct prctl_mm_map {# define PR_TIMER_CREATE_RESTORE_IDS_ON 1# define PR_TIMER_CREATE_RESTORE_IDS_GET 2 +#define PR_SET_FADE_ON_DEATH 78 +#define PR_GET_FADE_ON_DEATH 79 +#endif /* _LINUX_PRCTL_H */ kernel/sys.c +0
-0
kernel/sys.c SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3, · line 2665
Quoted 8 lines
kernel/sys.c +5
-0@@ -2658,6 +2658,22 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3,clear_bit (MMF_DISABLE_THP , &me->mm->flags);mmap_write_unlock (me->mm);break ; +case PR_GET_FADE_ON_DEATH : +if (arg2 || arg3 || arg4 || arg5) +return -EINVAL ; + error = !!test_bit (MMF_FADE_ON_DEATH , &me->mm->flags); +break ;
Is there a usecase for get?
kernel/sys.c SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3, · line 2666
Quoted 1 line
kernel/sys.c +1
-0+case PR_SET_FADE_ON_DEATH :
Could you just check the value prior to setting and just return if it's what you want? In which case, the setting is just change_bit(), and there probably isn't a need for a get?
Quoted 61 lines
kernel/sys.c +10
-0+if (arg3 || arg4 || arg5) +return -EINVAL ; +if (mmap_write_lock_killable (me->mm)) +return -EINTR ; +if (arg2) +set_bit (MMF_FADE_ON_DEATH , &me->mm->flags); +else +clear_bit (MMF_FADE_ON_DEATH , &me->mm->flags); +mmap_write_unlock (me->mm); +break ;case PR_MPX_ENABLE_MANAGEMENT :case PR_MPX_DISABLE_MANAGEMENT :/* No longer implemented: */ mm/huge_memory.c +10
-2@@ -2204,6 +2204,7 @@ static inline void zap_deposited_table(struct mm_struct *mm, pmd_t *pmd)int zap_huge_pmd (struct mmu_gather *tlb,struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr) { +bool drop_recency = false;pmd_t orig_pmd;spinlock_t *ptl;@@ -2260,13 +2261,20 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,add_mm_counter (tlb->mm,mm_counter_file (folio), -HPAGE_PMD_NR ); + drop_recency =zap_need_to_drop_recency (tlb->mm);/* * Use flush_needed to indicate whether the PMD entry * is present, instead of checking pmd_present() again. */ -if (flush_needed &&pmd_young (orig_pmd) && -likely (vma_has_recency (vma))) +if (flush_needed &&pmd_young (orig_pmd) && !drop_recency && +likely (vma_has_recency (vma)))folio_mark_accessed (folio); +/* +* Userspace explicitly marks recency to fade when the process +* dies; demote exclusive file folios to aid reclamation. +*/ +if (drop_recency && !folio_maybe_mapped_shared (folio)) +deactivate_file_folio (folio); }spin_unlock (ptl);mm/internal.h +1
-0@@ -11,6 +11,7 @@#include <linux/khugepaged.h> #include <linux/mm.h> #include <linux/mm_inline.h> +#include <linux/oom.h> #include <linux/pagemap.h> #include <linux/pagewalk.h> #include <linux/rmap.h>
mm/internal.h static inline int folio_nr_pages_mapped(const struct folio *folio) · line 139
Quoted 9 lines
mm/internal.h +6
-0@@ -130,6 +131,19 @@ static inline int folio_nr_pages_mapped(const struct folio *folio)return atomic_read (&folio->_nr_pages_mapped) &FOLIO_PAGES_MAPPED ; } +/* +* Returns true if the process attached to the mm is dying or undergoing +* OOM reaping, and its recency—explicitly marked by userspace—will also +* fade; otherwise, returns false. +*/ +static inline bool zap_need_to_drop_recency (struct mm_struct *mm)
This name is confusing. We are zapping the need to drop the recency? If this returns false, then the need to drop recency is false.. It is not very easy to read and harder to understand how it translates to the values it returns.
How about mm_has_exit_recency(), like vma_has_recency()? Or mmf_update_recency()?
mm/internal.h static inline int folio_nr_pages_mapped(const struct folio *folio) · line 141
Quoted 2 lines
mm/internal.h +2
-0+{ +if (!atomic_read (&mm->mm_users) ||check_stable_address_space (mm))
FYI, failed forks may also set the address space as unstable.
Quoted 21 lines
mm/internal.h +5
-0+return !!test_bit (MMF_FADE_ON_DEATH , &mm->flags); + +return false; +} +/* * Retrieve the first entry of a folio based on a provided entry within the * folio. We cannot rely on folio->swap as there is no guarantee that it has mm/memory.c +1
-0@@ -1505,6 +1505,7 @@ static __always_inline void zap_present_folio_ptes(struct mmu_gather *tlb, bool *force_flush, bool *force_break, bool *any_skipped) { struct mm_struct *mm = tlb->mm; +bool drop_recency = false;bool delay_rmap = false;if (!folio_test_anon (folio)) {
mm/memory.c static __always_inline void zap_present_folio_ptes(struct mmu_gather *tlb, · line 1523
Quoted 8 lines
mm/memory.c +4
-1@@ -1516,9 +1517,18 @@ static __always_inline void zap_present_folio_ptes(struct mmu_gather *tlb, *force_flush = true; } } -if (pte_young (ptent) &&likely (vma_has_recency (vma))) + + drop_recency =zap_need_to_drop_recency (mm); +if (pte_young (ptent) && !drop_recency && +likely (vma_has_recency (vma)))
I really don't like that you are calling an atomic_read() and two flag checks every time this block of code it executed. This must impact your performance?
How about this:
1. Check in unmap_vmas() that the range is 0 - ULONG_MAX, and if the OOM flag is set.
2. set a new zap_flags_t flag (mmf_update_recency, maybe?) if test_bit(MMF_FADE_ON_DEATH)
3. check zap_details->zap_flags if that bit is set in this function.
4. (hopefully) profit with better performance :)
Since this really is a zap flag, it fits to make it one. It also means that you will not need to check an atomic and will only check the one flag as apposed to two.
I think we can live with some user (probably syzbot) unmapping 0 - ULONG_MAX and incorrectly checking a flag and, in the very rare case of actually using this flag, does not do the correct LRU aging. If you unmap everything, we can be pretty confident that you will be on the exit path rather quickly.
mm/memory.c static __always_inline void zap_present_folio_ptes(struct mmu_gather *tlb, · line 1531
Quoted 8 lines
mm/memory.c +6
-0folio_mark_accessed (folio); rss[mm_counter (folio)] -= nr; +/* +* Userspace explicitly marks recency to fade when the process dies; +* demote exclusive file folios to aid reclamation. +*/ +if (drop_recency && !folio_maybe_mapped_shared (folio)) +deactivate_file_folio (folio);
Thanks,
Liam
Hi Liam,
I really appreciate your review—thank you!
Quoted 21 lines · Liam R. Howlett
Quoted 19 lines · Barry Song
From: Barry Song <[email protected]>
Currently, both
zap_pmdandzap_ptealways promote young file folios, regardless of whether the processes are dying.
However, in systems where the process recency fades upon dying, we may want to reverse this behavior. The goal is to reclaim the folios from the dying process as quickly as possible, allowing new processes to acquire memory ASAP.
For example, while Firefox is killed and LibreOffice is launched, activating Firefox's young file-backed folios makes it harder to reclaim memory that LibreOffice doesn't use at all.On systems like Android, processes are either explicitly stopped by user action or reaped due to OOM after being inactive for a long time. These processes are unlikely to restart in the near future. Rather than promoting their folios, we skip promoting and demote their exclusive folios so that memory can be reclaimed and made available for new user-facing processes.
Quoted 6 lines
Quoted 2 lines
Users possibly do not care about the recency of a dying process. However, we still need an explicit user indication to take this action.
Can you add why? It'd be nice to capture the reasons pointed out in v1 discussion as they seem important to why this isn't set as a default for all tasks.
Essentially, I took Johannes’ point (and to some extent David’s as well) to be that it behaves somewhat unpredictably in broader application scenarios—for example, when repeatedly executing a file in a script or restarting an application shortly after it exits.
Also, when a shared library is mapped by multiple processes, we might still want to retain recency information from a process that is exiting. So we might only want to do that only for exclusive folios.
This actually leads to two questions:
1. Are we confident that the recency of a dead process is no longer useful within a period of time?
2. Should we limit the optimization only to exclusive folios—for example, shared objects (.so files) that are specific to the exiting process?
For both questions, the answer seems to be yes.
Though in the first case—when we repeatedly restart the same application—the folios are likely still in the LRU and may still be hit even if we unconditionally demote them. But that's not guaranteed. So we likely need a userspace hint to eliminate the uncertainty.
Quoted 12 lines
Quoted 3 lines
Thus, we introduced a prctl to provide that necessary user-level hint as suggested by Johannes and David.
I'm not sure it really makes much of a difference if we update the lru or not in this case. Johannes point about this small change having unknown results for the larger community is certainly the best argument as to why we need this to be opt-in.
We should probably document it so that people can opt-in though :)
Quoted 1 line
Quoted 6 lines
Quoted 3 lines
We observed noticeable improvements in refaults, swap-ins, and swap-outs on a hooked Android kernel. More data for this specific version will follow.
Looking forward to the results. What happens when I kill my app and reopen it? (close all apps, open the one that was being annoying?)
I'm not sure I fully understand your question. In Android, we're primarily concerned with smooth app switching. For example, in a sequence like A → B → C → D → E, if we can quickly reclaim folios from dead processes, it helps us launch new (different) apps faster.
However, if we do A → kill A → start A → kill A → start A repeatedly, it’s likely not a problem because our memory can hold the same application. The issue arises when memory isn’t enough to hold A + B + C + D + E simultaneously.
I’m not overly concerned about repeatedly restarting the same application in Android. However, for wider scenarios across various industries, I’m uncertain.
Quoted 31 lines
Quoted 31 lines
Cc Baolin Wang <[email protected]>Cc David Hildenbrand <[email protected]>Cc Johannes Weiner <[email protected]>Cc Matthew Wilcox (Oracle) <[email protected]>Cc Oscar Salvador <[email protected]>Cc Ryan Roberts <[email protected]>Cc Zi Yan <[email protected]>Cc Lorenzo Stoakes <[email protected]>Cc "Liam R. Howlett" <[email protected]>Cc Vlastimil Babka <[email protected]>Cc Mike Rapoport <[email protected]>Cc Suren Baghdasaryan <[email protected]>Cc Michal Hocko <[email protected]>Signed-off-by Barry Song <[email protected]>---
-v2:
* add prctl as suggested by Johannes and David
* demote exclusive file folios ifdrop_recencycan apply -v1:
https://lore.kernel.org/linux-mm/[email protected]/include/linux/mm_types.h | 1 + include/uapi/linux/prctl.h | 3 +++ kernel/sys.c | 16 ++++++++++++++++ mm/huge_memory.c | 12 ++++++++++-- mm/internal.h | 14 ++++++++++++++ mm/memory.c | 12 +++++++++++- 6 files changed, 55 insertions(+), 3 deletions(-)
Quoted 12 lines
include/linux/mm_types.h enum { · line 1736
Quoted 9 lines
include/linux/mm_types.h +1
-0@@ -1733,6 +1733,7 @@ enum {* on NFS restore */ //#define MMF_EXE_FILE_CHANGED 18 /* see prctl_set_mm_exe_file() */ +#define MMF_FADE_ON_DEATH 18/* Recency is discarded on process exit */ Why is recency not in the MMF name? Why not
MMF_NO_RECENCYor something?
I included RECENCY in the name but found it too long. On the other hand, MMF_NO_RECENCY seems insufficient to convey the true meaning, since we do have recency—it’s just lost on death. So perhaps the original, longer names I considered are better: MMF_RECENCY_FADE_ON_DEATH or MMF_NO_RECENCY_ON_DEATH?
Quoted 2 lines
I guess we are back to no space in this flag.
Yes, it is 32 bits.
Quoted 23 lines
Quoted 23 lines
include/linux/mm_types.h +0
-0#define MMF_HAS_UPROBES 19/* has uprobes */ #define MMF_RECALC_UPROBES 20/* MMF_HAS_UPROBES can be wrong */ include/uapi/linux/prctl.h +3
-0@@ -364,4 +364,7 @@ struct prctl_mm_map {# define PR_TIMER_CREATE_RESTORE_IDS_ON 1# define PR_TIMER_CREATE_RESTORE_IDS_GET 2 +#define PR_SET_FADE_ON_DEATH 78 +#define PR_GET_FADE_ON_DEATH 79 +#endif /* _LINUX_PRCTL_H */ kernel/sys.c +0
-0@@ -2658,6 +2658,22 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3,clear_bit (MMF_DISABLE_THP , &me->mm->flags);mmap_write_unlock (me->mm);
Quoted 8 lines
kernel/sys.c SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3, · line 2665
Quoted 6 lines
kernel/sys.c +5
-0break ; +case PR_GET_FADE_ON_DEATH : +if (arg2 || arg3 || arg4 || arg5) +return -EINVAL ; + error = !!test_bit (MMF_FADE_ON_DEATH , &me->mm->flags); +break ;Is there a usecase for get?
Probably not. I was just trying to implement put/get for a pair. I’m happy to remove it if you feel it’s redundant.
Quoted 6 lines
kernel/sys.c SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3, · line 2666
Quoted 2 lines
kernel/sys.c +1
-0+case PR_SET_FADE_ON_DEATH :Could you just check the value prior to setting and just return if it's what you want? In which case, the setting is just
change_bit(), and there probably isn't a need for a get?
Ok.
Quoted 77 lines
Quoted 62 lines
kernel/sys.c +10
-0+if (arg3 || arg4 || arg5) +return -EINVAL ; +if (mmap_write_lock_killable (me->mm)) +return -EINTR ; +if (arg2) +set_bit (MMF_FADE_ON_DEATH , &me->mm->flags); +else +clear_bit (MMF_FADE_ON_DEATH , &me->mm->flags); +mmap_write_unlock (me->mm); +break ;case PR_MPX_ENABLE_MANAGEMENT :case PR_MPX_DISABLE_MANAGEMENT :/* No longer implemented: */ mm/huge_memory.c +10
-2@@ -2204,6 +2204,7 @@ static inline void zap_deposited_table(struct mm_struct *mm, pmd_t *pmd)int zap_huge_pmd (struct mmu_gather *tlb,struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr) { +bool drop_recency = false;pmd_t orig_pmd;spinlock_t *ptl;@@ -2260,13 +2261,20 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,add_mm_counter (tlb->mm,mm_counter_file (folio), -HPAGE_PMD_NR ); + drop_recency =zap_need_to_drop_recency (tlb->mm);/* * Use flush_needed to indicate whether the PMD entry * is present, instead of checking pmd_present() again. */ -if (flush_needed &&pmd_young (orig_pmd) && -likely (vma_has_recency (vma))) +if (flush_needed &&pmd_young (orig_pmd) && !drop_recency && +likely (vma_has_recency (vma)))folio_mark_accessed (folio); +/* +* Userspace explicitly marks recency to fade when the process +* dies; demote exclusive file folios to aid reclamation. +*/ +if (drop_recency && !folio_maybe_mapped_shared (folio)) +deactivate_file_folio (folio); }spin_unlock (ptl);mm/internal.h +1
-0@@ -11,6 +11,7 @@#include <linux/khugepaged.h> #include <linux/mm.h> #include <linux/mm_inline.h> +#include <linux/oom.h> #include <linux/pagemap.h> #include <linux/pagewalk.h> #include <linux/rmap.h> mm/internal.h static inline int folio_nr_pages_mapped(const struct folio *folio) · line 139
Quoted 9 lines
mm/internal.h +6
-0@@ -130,6 +131,19 @@ static inline int folio_nr_pages_mapped(const struct folio *folio)return atomic_read (&folio->_nr_pages_mapped) &FOLIO_PAGES_MAPPED ; } +/* +* Returns true if the process attached to the mm is dying or undergoing +* OOM reaping, and its recency—explicitly marked by userspace—will also +* fade; otherwise, returns false. +*/ +static inline bool zap_need_to_drop_recency (struct mm_struct *mm)This name is confusing. We are zapping the need to drop the recency? If this returns false, then the need to drop recency is false.. It is not very easy to read and harder to understand how it translates to the values it returns.
Quoted 2 lines
How about
mm_has_exit_recency(), likevma_has_recency()? Ormmf_update_recency()?
It seems mm_has_exit_recency() is good.
Quoted 25 lines
mm/internal.h static inline int folio_nr_pages_mapped(const struct folio *folio) · line 141
Quoted 3 lines
mm/internal.h +2
-0+{ +if (!atomic_read (&mm->mm_users) ||check_stable_address_space (mm))FYI, failed forks may also set the address space as unstable.
Quoted 19 lines
mm/internal.h +5
-0+return !!test_bit (MMF_FADE_ON_DEATH , &mm->flags); + +return false; +} +/* * Retrieve the first entry of a folio based on a provided entry within the * folio. We cannot rely on folio->swap as there is no guarantee that it has mm/memory.c +1
-0@@ -1505,6 +1505,7 @@ static __always_inline void zap_present_folio_ptes(struct mmu_gather *tlb, bool *force_flush, bool *force_break, bool *any_skipped) { struct mm_struct *mm = tlb->mm; +bool drop_recency = false;bool delay_rmap = false;
Quoted 15 lines
mm/memory.c static __always_inline void zap_present_folio_ptes(struct mmu_gather *tlb, · line 1523
Quoted 10 lines
mm/memory.c +4
-1if (!folio_test_anon (folio)) {@@ -1516,9 +1517,18 @@ static __always_inline void zap_present_folio_ptes(struct mmu_gather *tlb, *force_flush = true; } } -if (pte_young (ptent) &&likely (vma_has_recency (vma))) + + drop_recency =zap_need_to_drop_recency (mm); +if (pte_young (ptent) && !drop_recency && +likely (vma_has_recency (vma)))I really don't like that you are calling an
atomic_read()and two flag checks every time this block of code it executed. This must impact your performance?
Fair enough. That seems like a valid point to consider regarding atomic operations.
Quoted 13 lines
How about this:
1. Check inunmap_vmas()that the range is 0 -ULONG_MAX, and if the OOM flag is set.
2. set a newzap_flags_tflag (mmf_update_recency, maybe?) iftest_bit(MMF_FADE_ON_DEATH)
3. checkzap_details->zap_flagsif that bit is set in this function.
4. (hopefully) profit with better performance :)Since this really is a zap flag, it fits to make it one. It also means that you will not need to check an atomic and will only check the one flag as apposed to two.
Quoted 5 lines
I think we can live with some user (probably syzbot) unmapping 0 -
ULONG_MAXand incorrectly checking a flag and, in the very rare case of actually using this flag, does not do the correct LRU aging. If you unmap everything, we can be pretty confident that you will be on the exit path rather quickly.
Good idea—let me give this a try.
Quoted 10 lines
mm/memory.c static __always_inline void zap_present_folio_ptes(struct mmu_gather *tlb, · line 1531
Quoted 9 lines
mm/memory.c +6
-0folio_mark_accessed (folio); rss[mm_counter (folio)] -= nr; +/* +* Userspace explicitly marks recency to fade when the process dies; +* demote exclusive file folios to aid reclamation. +*/ +if (drop_recency && !folio_maybe_mapped_shared (folio)) +deactivate_file_folio (folio);
Quoted 3 lines
Thanks,
Liam
Thanks
Barry
Quoted 2 lines · Barry Song
Hi Liam,
I really appreciate your review—thank you!
Thanks.
This came up when discussing another policy type control to an entire mm [1].
It looks like the consensus is that a new system call for memory specific controls might be the best way forward, and your requirement fits well into this idea.
I'd be interested in hearing what you think on this plan.
[1]. https://lore.kernel.org/all/[email protected]/
Regards,
Liam