mm: MM owner tracking for large folios (!hugetlb) + CONFIG_NO_PAGE_MAPCOUNT
Subsystems (19)
v2 cover mm: MM owner tracking for large folios (!hugetlb) + CONFIG_NO_PAGE_MAPCOUNT
This patch series is about to haunt me in my dreams, and the time spent on this is ridiculous ... anyhow, here goes a new version that now also supports 32bit (I wish we would have dynamically allocated "struct folio" already ...) and always enables the new tracking with CONFIG_TRANSPARENT_HUGEPAGE.
Let's add an "easy" way to decide -- without false positives, without page-mapcounts and without page table/rmap scanning -- whether a large folio is "certainly mapped exclusively" into a single MM, or whether it "maybe mapped shared" into multiple MMs.
Use that information to implement Copy-on-Write reuse, to convert folio_likely_mapped_shared() to folio_maybe_mapped_share(), and to introduce a kernel config option that let's us not use+maintain per-page mapcounts in large folios anymore.
The bigger picture was presented at LSF/MM [1].
This series is effectively a follow-up on my early work [2], which implemented a more precise, but also more complicated, way to identify whether a large folio is "mapped shared" into multiple MMs or "mapped exclusively" into a single MM.
1 Patch Organization
====================
Patch #1 -> #6: make more room in order-1 folios, so we have two
"unsigned long" available for our purposes
Patch #7 -> #11: preparations
Patch #12: MM owner tracking for large folios
Patch #13: COW reuse for PTE-mapped anon THP
Patch #14: folio_maybe_mapped_shared()
Patch #15 -> #20: introduce and implement CONFIG_NO_PAGE_MAPCOUNT
2 MM owner tracking
===================
We assign each MM a unique ID ("MM ID"), to be able to squeeze more information in our folios. On 32bit we use 15-bit IDs, on 64bit we use 31-bit IDs.
For each large folios, we now store two MM-ID+mapcount ("slot") combinations:
* mm0_id + mm0_mapcount
* mm1_id + mm1_mapcount
On 32bit, we use a 16-bit per-MM mapcount, on 64bit an ordinary 32bit mapcount. This way, we require 2x "unsigned long" on 32bit and 64bit for both slots.
Paired with the large mapcount, we can reliably identify whether one of these MMs is the current owner (-> owns all mappings) or even holds all folio references (-> owns all mappings, and all references are from mappings).
As long as only two MMs map folio pages at a time, we can reliably and precisely identify whether a large folio is "mapped shared" or "mapped exclusively".
Any additional MM that starts mapping the folio while there are no free slots becomes an "untracked MM". If one such "untracked MM" is the last one mapping a folio exclusively, we will *not* detect the folio as "mapped exclusively" but instead as "maybe mapped shared". (exception:
only a single mapping remains)
So that's where the approach gets imprecise.
For now, we use a bit-spinlock to sync the large mapcount + slots, and make sure we do keep the machinery fast, to not degrade (un)map performance
drastically: for example, we make sure to only use a single atomic (when grabbing the bit-spinlock), like we would already perform when updating the large mapcount.
3 CONFIG_NO_PAGE_MAPCOUNT
=========================
patch #15 -> #20 spell out and document what exactly is affected when not maintaining the per-page mapcounts in large folios anymore.
Most importantly, as we cannot maintain folio->_nr_pages_mapped anymore when (un)mapping pages, we'll account a complete folio as mapped if a single page is mapped. In addition, we'll not detect partially mapped anonymous folios as such in all cases yet.
Likely less relevant changes include that we might now under-estimate the USS (Unique Set Size) of a process, but never over-estimate it.
The goal is to make CONFIG_NO_PAGE_MAPCOUNT the default at some point, to then slowly make it the only option, as we learn about real-life impacts and possible ways to mitigate them.
4 Performance
=============
Detailed performance numbers were included in v1 [4], and not that much changed between v1 and v2.
I did plenty of measurements on different systems in the meantime, that all revealed slightly different results.
The pte-mapped-folio micro-benchmarks are fairly sensitive to code layout changes on some systems. Especially the fork() benchmark started being more-shaky-than-before on recent kernels for some reason.
In summary, with my micro-benchmarks:
* Small folios are not impacted.
* CoW performance seems to be mostly unchanged across all folios sizes.
* CoW reuse performance of large folios now matches CoW reuse performance of small folios, because we now actually implement the CoW reuse optimization. On an Intel Xeon Silver 4210R I measured a ~65% reduction in runtime, on an arm64 system I measured ~54% reduction.
* munmap() performance improves with CONFIG_NO_PAGE_MAPCOUNT. I saw double-digit % reduction (up to ~30% on an Intel Xeon Silver 4210R and up to ~70% on an AmpereOne A192-32X) with larger folios. The larger the folios, the larger the performance improvement.
* munmao() performance very slightly (couple percent) degrades without CONFIG_NO_PAGE_MAPCOUNT for smaller folios. For larger folios, there seems to be no change at all.
* fork() performance improves with CONFIG_NO_PAGE_MAPCOUNT. I saw double-digit % reduction (up to ~20% on an Intel Xeon Silver 4210R and up to ~10% on an AmpereOne A192-32X) with larger folios. The larger the folios, the larger the performance improvement.
* While fork() performance without CONFIG_NO_PAGE_MAPCOUNT seems to be almost unchanged on some systems, I saw some degradation for smaller folios on the AmpereOne A192-32X. I did not investigate the details yet, but I suspect code layout changes or suboptimal code placement / inlining.
I'm not to worried about the fork() micro-benchmarks for smaller folios given how shaky the results are lately and by how much we improved fork() performance recently.
I also ran case-anon-cow-rand and case-anon-cow-seq part of vm-scalability, to assess the scalability and the impact of the bit-spinlock. My measurements on a two 2-socket 10-core Intel Xeon Silver 4210R CPU revealed no significant changes.
Similarly, running these benchmarks with 2 MiB THPs enabled on the AmpereOne A192-32X with 192 cores, I got < 1% difference with < 1% stdev, which is nice.
So far, I did not get my hands on a similarly large system with multiple sockets.
I found no other fitting scalability benchmarks that seem to really hammer on concurrent mapping/unmapping of large folio pages like case-anon-cow-seq does.
5 Concerns
==========
5.1 Bit spinlock
----------------
I'm not quite happy about the bit-spinlock, but so far it does not seem to affect scalability in my measurements.
If it ever becomes a problem we could either investigate improving the locking, or simply stopping the MM tracking once there are "too many mappings" and simply assume that the folio is "mapped shared" until it was freed.
This would be similar (but slightly different) to the "0,1,2,stopped" counting idea Willy had at some point. Adding that logic to "stop tracking" adds more code to the hot path, so I avoided that for now.
5.2 folio_maybe_mapped_shared()
-------------------------------
I documented the change from folio_likely_mapped_shared() to folio_maybe_mapped_shared() quite extensively. If we run into surprises, I have some ideas on how to resolve them. For now, I think we should be fine.
5.3 Added code to map/unmap hot path
------------------------------------
So far, it looks like the added code on the rmap hot path does not really seem to matter much in the bigger picture. I'd like to further reduce it (and possibly improve fork() performance further), but I don't easily see how right now. Well, and I am out of puff :)
Having that said, alternatives I considered (e.g., per-MM per-folio mapcount) would add a lot more overhead to these hot paths.
6 Future Work
=============
6.1 Large mapcount
------------------
It would be very handy if the large mapcount would count how often folio pages are actually mapped into page tables: a PMD on x86-64 would count 512 times. Calculating the average per-page mapcount will be easy, and remapping (PMD->PTE) folios would get even faster.
That would also remove the need for the entire mapcount (except for PMD-sized folios for memory statistics reasons ...), and allow for mapping folios larger than PMDs (e.g., 4 MiB) easily.
The downside is that we maybe would also have to take the same number of folio references to make our folio_mapcount() == folio_ref_count() work. I think it should be possible (user space could trigger many PTE mappings already), but we be a bit more careful about possible mapcount/refcount overflows. (e.g., fail mapping a folio if the mapcount exceeds a certain threshold)
Maybe some day we'll have a 64bit refcount+mapcount.
6.2 hugetlb
-----------
I'd love to make use of the same tracking also for hugetlb.
The real problem is PMD table sharing: getting a page mapped by MM X and unmapped by MM Y will not work. With mshare, that problem should not exist (all mapping/unmapping will be routed through the mshare MM).
7 Version Updates
=================
I did a bunch of cross-compiles and quite some testing on i386, x86-64 and arm64. The build bots were very helpful as well.
To keep the CC list short, adding only relevant subsystem maintainers (CCed on all patches, sorry :) ).
v1 -> v2:
* 32bit support. It would all be easier if we would already allocate "struct folio" dynamically, but fortunately when we manage to do that, it will just clean that part up again. For now, we have to relocate in "struct folio" the _pincount and _entire_mapcount on 32bit, and the hugetlb data unconditionally.
* "mm/rmap: basic MM owner tracking for large folios (!hugetlb)" -> Now unconditionally enabled with CONFIG_TRANSPARENT_HUGEPAGE -> Some changes to slot handling to handle some edge cases in a better way.
-> Reworked the way flags are stored, in light of 32bit support.
* "mm: convert folio_likely_mapped_shared() to folio_maybe_mapped_shared()" -> Use the new logic always such that we can rename the function
* A bunch of cleanups/simplifications
David Hildenbrand (20):
mm: factor out large folio handling from folio_order() into
folio_large_order()
mm: factor out large folio handling from folio_nr_pages() into
folio_large_nr_pages()
mm: let _folio_nr_pages overlay memcg_data in first tail page
mm: move hugetlb specific things in folio to page[3]
mm: move _pincount in folio to page[2] on 32bit
mm: move _entire_mapcount in folio to page[2] on 32bit mm/rmap: pass dst_vma to folio_dup_file_rmap_pte() and friends mm/rmap: pass vma to __folio_add_rmap()
mm/rmap: abstract large mapcount operations for large folios
(!hugetlb)
bit_spinlock: __always_inline (un)lock functions
mm/rmap: use folio_large_nr_pages() in add/remove functions mm/rmap: basic MM owner tracking for large folios (!hugetlb)
mm: Copy-on-Write (COW) reuse support for PTE-mapped THP
mm: convert folio_likely_mapped_shared() to
folio_maybe_mapped_shared()
mm: CONFIG_NO_PAGE_MAPCOUNT to prepare for not maintain per-page
mapcounts in large folios
fs/proc/page: remove per-page mapcount dependency for /proc/kpagecount
(CONFIG_NO_PAGE_MAPCOUNT)
fs/proc/task_mmu: remove per-page mapcount dependency for
PM_MMAP_EXCLUSIVE (CONFIG_NO_PAGE_MAPCOUNT)
fs/proc/task_mmu: remove per-page mapcount dependency for "mapmax"
(CONFIG_NO_PAGE_MAPCOUNT)
fs/proc/task_mmu: remove per-page mapcount dependency for
smaps/smaps_rollup (CONFIG_NO_PAGE_MAPCOUNT)
mm: stop maintaining the per-page mapcount of large folios
(CONFIG_NO_PAGE_MAPCOUNT)
.../admin-guide/cgroup-v1/memory.rst | 4 + Documentation/admin-guide/cgroup-v2.rst | 10 +- Documentation/admin-guide/mm/pagemap.rst | 16 +- Documentation/filesystems/proc.rst | 28 +- Documentation/mm/transhuge.rst | 39 ++- fs/proc/internal.h | 39 +++ fs/proc/page.c | 19 +- fs/proc/task_mmu.c | 39 ++- include/linux/bit_spinlock.h | 8 +- include/linux/mm.h | 93 +++--- include/linux/mm_types.h | 110 +++++-- include/linux/page-flags.h | 4 + include/linux/rmap.h | 270 ++++++++++++++++-- kernel/fork.c | 36 +++ mm/Kconfig | 22 ++ mm/debug.c | 10 +- mm/gup.c | 8 +- mm/huge_memory.c | 20 +- mm/hugetlb.c | 1 - mm/internal.h | 20 +- mm/khugepaged.c | 8 +- mm/madvise.c | 6 +- mm/memory.c | 96 ++++++- mm/mempolicy.c | 8 +- mm/migrate.c | 7 +- mm/mprotect.c | 2 +- mm/page_alloc.c | 50 +++- mm/page_owner.c | 2 +- mm/rmap.c | 118 ++++++-- 29 files changed, 903 insertions(+), 190 deletions(-)
base-commit: f7ed46277aaa8f848f18959ff68469f5186ba87c
Signature · 1 line
2.48.1
PATCH v2 1/20 mm: factor out large folio handling from folio_order() into folio_large_order()
Let's factor it out into a simple helper function. This helper will also come in handy when working with code where we know that our folio is large.
Maybe in the future we'll have the order readily available for small and large folios; in that case, folio_large_order() would simply translate to folio_order().
--- include/linux/mm.h | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)
include/linux/mm.h +9 -4
@@ -1194,6 +1194,11 @@ struct inode;extern void prep_compound_page (struct page *page,unsigned int order); +static inline unsigned int folio_large_order (const struct folio *folio) +{ +return folio->_flags_1 &0xff ; +} +/* * compound_order() can be called without holding a reference, which means * that niceties like page_folio() don't work. These callers should be
@@ -1207,7 +1212,7 @@ static inline unsigned int compound_order(struct page *page)if (!test_bit (PG_head, &folio->flags))return 0 ; -return folio->_flags_1 &0xff ; +return folio_large_order (folio); }/**
@@ -1223,7 +1228,7 @@ static inline unsigned int folio_order(const struct folio *folio) {if (!folio_test_large (folio))return 0 ; -return folio->_flags_1 &0xff ; +return folio_large_order (folio); }#include <linux/huge_mm.h>
@@ -2139,7 +2144,7 @@ static inline long folio_nr_pages(const struct folio *folio)#ifdef CONFIG_64BIT return folio->_folio_nr_pages;#else -return 1L << (folio->_flags_1 &0xff ); +return 1L <<folio_large_order (folio);#endif }
@@ -2164,7 +2169,7 @@ static inline unsigned long compound_nr(struct page *page)#ifdef CONFIG_64BIT return folio->_folio_nr_pages;#else -return 1L << (folio->_flags_1 &0xff ); +return 1L <<folio_large_order (folio);#endif }
Signature · 1 line
2.48.1
PATCH v2 2/20 mm: factor out large folio handling from folio_nr_pages() into folio_large_nr_pages()
In reply to David HildenbrandLet's factor it out into a simple helper function. This helper will also come in handy when working with code where we know that our folio is large.
While at it, let's consistently return a "long" value from all these similar functions. Note that we cannot use "unsigned int" (even though _folio_nr_pages is of that type), because it would break some callers that do stuff like "-folio_nr_pages()". Both "int" or "unsigned long" would work as well.
Maybe in the future we'll have the nr_pages readily available for all large folios, maybe even for small folios, or maybe for none.
--- include/linux/mm.h | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-)
include/linux/mm.h +16 -12
@@ -1199,6 +1199,18 @@ static inline unsigned int folio_large_order(const struct folio *folio)return folio->_flags_1 &0xff ; } +#ifdef CONFIG_64BIT +static inline long folio_large_nr_pages (const struct folio *folio) +{ +return folio->_folio_nr_pages; +} +#else +static inline long folio_large_nr_pages (const struct folio *folio) +{ +return 1L <<folio_large_order (folio); +} +#endif +/* * compound_order() can be called without holding a reference, which means * that niceties like page_folio() don't work. These callers should be
@@ -2141,11 +2153,7 @@ static inline long folio_nr_pages(const struct folio *folio) {if (!folio_test_large (folio))return 1 ; -#ifdef CONFIG_64BIT -return folio->_folio_nr_pages; -#else -return 1L <<folio_large_order (folio); -#endif +return folio_large_nr_pages (folio); }/* Only hugetlbfs can allocate folios larger than MAX_ORDER */
@@ -2160,24 +2168,20 @@ static inline long folio_nr_pages(const struct folio *folio)* page. compound_nr() can be called on a tail page, and is defined to * return 1 in that case. */ -static inline unsigned long compound_nr (struct page *page) +static inline long compound_nr (struct page *page) {struct folio *folio = (struct folio *)page;if (!test_bit (PG_head, &folio->flags))return 1 ; -#ifdef CONFIG_64BIT -return folio->_folio_nr_pages; -#else -return 1L <<folio_large_order (folio); -#endif +return folio_large_nr_pages (folio); }/** * thp_nr_pages - The number of regular pages in this huge page. * @page: The head page of a huge page. */ -static inline int thp_nr_pages (struct page *page) +static inline long thp_nr_pages (struct page *page) {return folio_nr_pages ((struct folio *)page); }
Signature · 1 line
2.48.1
PATCH v2 3/20 mm: let _folio_nr_pages overlay memcg_data in first tail page
In reply to David HildenbrandLet's free up some more of the "unconditionally available on 64BIT" space in order-1 folios by letting _folio_nr_pages overlay memcg_data in the first tail page (second folio page). Consequently, we have the optimization now whenever we have CONFIG_MEMCG, independent of 64BIT.
We have to make sure that page->memcg on tail pages does not return "surprises". page_memcg_check() already properly refuses PageTail(). Let's do that earlier in print_page_owner_memcg() to avoid printing wrong "Slab cache page" information. No other code should touch that field on tail pages of compound pages.
Reset the "_nr_pages" to 0 when splitting folios, or when freeing them back to the buddy (to avoid false page->memcg_data "bad page" reports).
Note that in __split_huge_page(), folio_nr_pages() would stop working already as soon as we start messing with the subpages.
Most kernel configs should have at least CONFIG_MEMCG enabled, even if disabled at runtime. 64byte "struct memmap" is what we usually have on 64BIT.
While at it, rename "_folio_nr_pages" to "_nr_pages".
Hopefully memdescs / dynamically allocating "strut folio" in the future will further clean this up, e.g., making _nr_pages available in all configs and maybe even in small folios. Doing that should be fairly easy on top of this change.
--- include/linux/mm.h | 4 ++-- include/linux/mm_types.h | 30 ++++++++++++++++++++++-------- mm/huge_memory.c | 16 +++++++++++++--- mm/internal.h | 4 ++-- mm/page_alloc.c | 6 +++++- mm/page_owner.c | 2 +- 6 files changed, 45 insertions(+), 17 deletions(-)
include/linux/mm.h +2 -2
@@ -1199,10 +1199,10 @@ static inline unsigned int folio_large_order(const struct folio *folio)return folio->_flags_1 &0xff ; } -#ifdef CONFIG_64BIT +#ifdef NR_PAGES_IN_LARGE_FOLIO static inline long folio_large_nr_pages (const struct folio *folio) { -return folio->_folio_nr_pages; +return folio->_nr_pages; }#else static inline long folio_large_nr_pages (const struct folio *folio)
include/linux/mm_types.h +22 -8
@@ -287,6 +287,11 @@ typedef struct {unsigned long val; } swp_entry_t; +#if defined(CONFIG_MEMCG ) || defined(CONFIG_SLAB_OBJ_EXT ) +/* We have some extra room after the refcount in tail pages. */ +#define NR_PAGES_IN_LARGE_FOLIO +#endif +/** * struct folio - Represents a contiguous set of bytes. * @flags: Identical to the page flags.
@@ -312,7 +317,7 @@ typedef struct {* @_large_mapcount: Do not use directly, call folio_mapcount(). * @_nr_pages_mapped: Do not use outside of rmap and debug code. * @_pincount: Do not use directly, call folio_maybe_dma_pinned(). -* @_folio_nr_pages: Do not use directly, call folio_nr_pages(). +* @_nr_pages: Do not use directly, call folio_nr_pages(). * @_hugetlb_subpool: Do not use directly, use accessor in hugetlb.h. * @_hugetlb_cgroup: Do not use directly, use accessor in hugetlb_cgroup.h. * @_hugetlb_cgroup_rsvd: Do not use directly, use accessor in hugetlb_cgroup.h.
@@ -377,13 +382,20 @@ struct folio {unsigned long _flags_1;unsigned long _head_1;/* public: */ -atomic_t _large_mapcount; -atomic_t _entire_mapcount; -atomic_t _nr_pages_mapped; -atomic_t _pincount; -#ifdef CONFIG_64BIT -unsigned int _folio_nr_pages; -#endif +union { +struct { +atomic_t _large_mapcount; +atomic_t _entire_mapcount; +atomic_t _nr_pages_mapped; +atomic_t _pincount; + }; +unsigned long _usable_1[4 ]; + }; +atomic_t _mapcount_1; +atomic_t _refcount_1; +#ifdef NR_PAGES_IN_LARGE_FOLIO +unsigned int _nr_pages; +#endif /* NR_PAGES_IN_LARGE_FOLIO */ /* private: the union with struct page is transitional */ };struct page __page_1;
@@ -435,6 +447,8 @@ FOLIO_MATCH(_last_cpupid, _last_cpupid); offsetof(struct page, pg) + sizeof(struct page))FOLIO_MATCH (flags, _flags_1);FOLIO_MATCH (compound_head, _head_1); +FOLIO_MATCH (_mapcount, _mapcount_1); +FOLIO_MATCH (_refcount, _refcount_1);#undef FOLIO_MATCH#define FOLIO_MATCH (pg, fl) \ static_assert(offsetof(struct folio, fl) == \
mm/huge_memory.c +13 -3
@@ -3307,9 +3307,10 @@ bool can_split_folio(struct folio *folio, int caller_pins, int *pextra_pins)* It splits @folio into @new_order folios and copies the @folio metadata to * all the resulting folios. */ -static void __split_folio_to_order (struct folio *folio,int new_order) +static void __split_folio_to_order (struct folio *folio,int old_order, +int new_order) { -long nr_pages =folio_nr_pages (folio); +long nr_pages =1 << old_order;long new_nr_pages =1 << new_order;long index;
@@ -3529,12 +3530,21 @@ static int __split_unmapped_folio(struct folio *folio, int new_order, } } +/* +* Reset any memcg data overlay in the tail pages. +* folio_nr_pages() is unreliable until prep_compound_page() +* was called again. +*/ +#ifdef NR_PAGES_IN_LARGE_FOLIO + folio->_nr_pages =0 ; +#endif +/* complete memcg works before add pages to LRU */ split_page_memcg (&folio->page, old_order, split_order);split_page_owner (&folio->page, old_order, split_order);pgalloc_tag_split (folio, old_order, split_order); -__split_folio_to_order (folio, split_order); +__split_folio_to_order (folio, old_order, split_order); after_split:/*
mm/internal.h +2 -2
@@ -684,8 +684,8 @@ static inline void folio_set_order(struct folio *folio, unsigned int order)return ; folio->_flags_1 = (folio->_flags_1 & ~0xffUL ) | order; -#ifdef CONFIG_64BIT - folio->_folio_nr_pages =1U << order; +#ifdef NR_PAGES_IN_LARGE_FOLIO + folio->_nr_pages =1U << order;#endif }
mm/page_alloc.c +5 -1
@@ -1174,8 +1174,12 @@ __always_inline bool free_pages_prepare(struct page *page,if (unlikely (order)) {int i; -if (compound) +if (compound) { page[1 ].flags &= ~PAGE_FLAGS_SECOND ; +#ifdef NR_PAGES_IN_LARGE_FOLIO + folio->_nr_pages =0 ; +#endif + }for (i =1 ; i < (1 << order); i++) {if (compound) bad +=free_tail_page_prepare (page, page + i);
mm/page_owner.c +1 -1
@@ -507,7 +507,7 @@ static inline int print_page_owner_memcg(char *kbuf, size_t count, int ret,rcu_read_lock (); memcg_data =READ_ONCE (page->memcg_data); -if (!memcg_data) +if (!memcg_data ||PageTail (page)) goto out_unlock;if (memcg_data &MEMCG_DATA_OBJEXTS )
Signature · 1 line
2.48.1
PATCH v2 4/20 mm: move hugetlb specific things in folio to page[3]
In reply to David HildenbrandLet's just move the hugetlb specific stuff to a separate page, and stop letting it overlay other fields for now.
This frees up some space in page[2], which we will use on 32bit to free up some space in page[1]. While we could move these things to page[3] instead, it's cleaner to just move the hugetlb specific things out of the way and pack the core-folio stuff as tight as possible. ... and we can minimize the work required in dump_folio.
We can now avoid re-initializing &folio->_deferred_list in hugetlb code.
Hopefully dynamically allocating "strut folio" in the future will further clean this up.
--- include/linux/mm_types.h | 27 +++++++++++++++++---------- mm/hugetlb.c | 1 - mm/page_alloc.c | 5 +++++ 3 files changed, 22 insertions(+), 11 deletions(-)
include/linux/mm_types.h +17 -10
@@ -405,20 +405,23 @@ struct folio {unsigned long _flags_2;unsigned long _head_2;/* public: */ -void *_hugetlb_subpool; -void *_hugetlb_cgroup; -void *_hugetlb_cgroup_rsvd; -void *_hugetlb_hwpoison; +struct list_head _deferred_list;/* private: the union with struct page is transitional */ }; +struct page __page_2; + }; +union {struct { -unsigned long _flags_2a; -unsigned long _head_2a; +unsigned long _flags_3; +unsigned long _head_3;/* public: */ -struct list_head _deferred_list; +void *_hugetlb_subpool; +void *_hugetlb_cgroup; +void *_hugetlb_cgroup_rsvd; +void *_hugetlb_hwpoison;/* private: the union with struct page is transitional */ }; -struct page __page_2; +struct page __page_3; }; };
@@ -455,8 +458,12 @@ FOLIO_MATCH(_refcount, _refcount_1); offsetof(struct page, pg) + 2 * sizeof(struct page))FOLIO_MATCH (flags, _flags_2);FOLIO_MATCH (compound_head, _head_2); -FOLIO_MATCH (flags, _flags_2a); -FOLIO_MATCH (compound_head, _head_2a); +#undef FOLIO_MATCH +#define FOLIO_MATCH (pg, fl) \ + static_assert(offsetof(struct folio, fl) == \ + offsetof(struct page, pg) + 3 * sizeof(struct page)) +FOLIO_MATCH (flags, _flags_3); +FOLIO_MATCH (compound_head, _head_3);#undef FOLIO_MATCH/**
mm/hugetlb.c +0 -1
@@ -1646,7 +1646,6 @@ static void __update_and_free_hugetlb_folio(struct hstate *h,folio_ref_unfreeze (folio,1 ); -INIT_LIST_HEAD (&folio->_deferred_list);hugetlb_free_folio (folio); }
mm/page_alloc.c +5 -0
@@ -971,6 +971,11 @@ static int free_tail_page_prepare(struct page *head_page, struct page *page) goto out; }break ; +case 3 : +/* the third tail page: hugetlb specifics overlap ->mappings */ +if (IS_ENABLED (CONFIG_HUGETLB_PAGE )) +break ; + fallthrough;default :if (page->mapping !=TAIL_MAPPING ) {bad_page (page,"corrupted mapping in tail page" );
Signature · 1 line
2.48.1
PATCH v2 5/20 mm: move _pincount in folio to page[2] on 32bit
In reply to David HildenbrandLet's free up some space on 32bit in page[1] by moving the _pincount to page[2].
For order-1 folios (never anon folios!) on 32bit, we will now also use the GUP_PIN_COUNTING_BIAS approach. A fully-mapped order-1 folio requires 2 references. With GUP_PIN_COUNTING_BIAS being 1024, we'd detect such folios as "maybe pinned" with 512 full mappings, instead of 1024 for order-0. As anon folios are out of the picture (which are the most relevant users of checking for pinnings on *mapped* pages) and we are talking about 32bit, this is not expected to cause any trouble.
In __dump_page(), copy one additional folio page if we detect a folio with an order > 1, so we can dump the pincount on order > 1 folios reliably.
Note that THPs on 32bit are not particularly common (and we don't care too much about performance), but we want to keep it working reliably, because likely we want to use large folios there as well in the future, independent of PMD leaf support.
Once we dynamically allocate "struct folio", fortunately the 32bit specifics will likely go away again; even small folios could then have a pincount and folio_has_pincount() would essentially always return "true".
--- include/linux/mm.h | 11 +++++++++-- include/linux/mm_types.h | 5 +++++ mm/debug.c | 10 +++++++++- mm/gup.c | 8 ++++---- mm/internal.h | 3 ++- mm/page_alloc.c | 14 +++++++++++--- 6 files changed, 40 insertions(+), 11 deletions(-)
include/linux/mm.h +9 -2
@@ -2004,6 +2004,13 @@ static inline struct folio *pfn_folio(unsigned long pfn)return page_folio (pfn_to_page (pfn)); } +static inline bool folio_has_pincount (const struct folio *folio) +{ +if (IS_ENABLED (CONFIG_64BIT )) +return folio_test_large (folio); +return folio_order (folio) >1 ; +} +/** * folio_maybe_dma_pinned - Report if a folio may be pinned for DMA. * @folio: The folio.
@@ -2020,7 +2027,7 @@ static inline struct folio *pfn_folio(unsigned long pfn)* get that many refcounts, and b) all the callers of this routine are * expected to be able to deal gracefully with a false positive. * -* For large folios, the result will be exactly correct. That's because +* For most large folios, the result will be exactly correct. That's because * we have more tracking data available: the _pincount field is used * instead of the GUP_PIN_COUNTING_BIAS scheme. *
@@ -2031,7 +2038,7 @@ static inline struct folio *pfn_folio(unsigned long pfn)*/ static inline bool folio_maybe_dma_pinned (struct folio *folio) { -if (folio_test_large (folio)) +if (folio_has_pincount (folio))return atomic_read (&folio->_pincount) >0 ;/*
include/linux/mm_types.h +5 -0
@@ -387,7 +387,9 @@ struct folio {atomic_t _large_mapcount;atomic_t _entire_mapcount;atomic_t _nr_pages_mapped; +#ifdef CONFIG_64BIT atomic_t _pincount; +#endif /* CONFIG_64BIT */ };unsigned long _usable_1[4 ]; };
@@ -406,6 +408,9 @@ struct folio {unsigned long _head_2;/* public: */ struct list_head _deferred_list; +#ifndef CONFIG_64BIT +atomic_t _pincount; +#endif /* !CONFIG_64BIT */ /* private: the union with struct page is transitional */ };struct page __page_2;
mm/debug.c +9 -1
@@ -79,12 +79,17 @@ static void __dump_folio(struct folio *folio, struct page *page,folio_ref_count (folio), mapcount, mapping, folio->index + idx, pfn);if (folio_test_large (folio)) { +int pincount =0 ; + +if (folio_has_pincount (folio)) + pincount =atomic_read (&folio->_pincount); +pr_warn ("head: order:%u mapcount:%d entire_mapcount:%d nr_pages_mapped:%d pincount:%d\n" ,folio_order (folio),folio_mapcount (folio),folio_entire_mapcount (folio),folio_nr_pages_mapped (folio), -atomic_read (&folio->_pincount)); + pincount); }#ifdef CONFIG_MEMCG
@@ -146,6 +151,9 @@ static void __dump_page(const struct page *page)if (idx <MAX_FOLIO_NR_PAGES ) {memcpy (&folio, foliop,2 *sizeof (struct page )); nr_pages =folio_nr_pages (&folio); +if (nr_pages >1 ) +memcpy (&folio.__page_2, &foliop->__page_2, +sizeof (struct page )); foliop = &folio; }
mm/gup.c +4 -4
@@ -109,7 +109,7 @@ static void gup_put_folio(struct folio *folio, int refs, unsigned int flags)if (is_zero_folio (folio))return ;node_stat_mod_folio (folio,NR_FOLL_PIN_RELEASED , refs); -if (folio_test_large (folio)) +if (folio_has_pincount (folio))atomic_sub (refs, &folio->_pincount);else refs *=GUP_PIN_COUNTING_BIAS ;
@@ -164,7 +164,7 @@ int __must_check try_grab_folio(struct folio *folio, int refs,* Increment the normal page refcount field at least once, * so that the page really is pinned. */ -if (folio_test_large (folio)) { +if (folio_has_pincount (folio)) {folio_ref_add (folio, refs);atomic_add (refs, &folio->_pincount); }else {
@@ -223,7 +223,7 @@ void folio_add_pin(struct folio *folio)* page refcount field at least once, so that the page really is * pinned. */ -if (folio_test_large (folio)) { +if (folio_has_pincount (folio)) {WARN_ON_ONCE (atomic_read (&folio->_pincount) <1 );folio_ref_inc (folio);atomic_inc (&folio->_pincount);
@@ -575,7 +575,7 @@ static struct folio *try_grab_folio_fast(struct page *page, int refs,* is pinned. That's why the refcount from the earlier * try_get_folio() is left intact. */ -if (folio_test_large (folio)) +if (folio_has_pincount (folio))atomic_add (refs, &folio->_pincount);else folio_ref_add(folio,
mm/internal.h +2 -1
@@ -723,7 +723,8 @@ static inline void prep_compound_head(struct page *page, unsigned int order)atomic_set (&folio->_large_mapcount,-1 );atomic_set (&folio->_entire_mapcount,-1 );atomic_set (&folio->_nr_pages_mapped,0 ); -atomic_set (&folio->_pincount,0 ); +if (IS_ENABLED (CONFIG_64BIT ) || order >1 ) +atomic_set (&folio->_pincount,0 );if (order >1 )INIT_LIST_HEAD (&folio->_deferred_list); }
mm/page_alloc.c +11 -3
@@ -959,9 +959,11 @@ static int free_tail_page_prepare(struct page *head_page, struct page *page)bad_page (page,"nonzero nr_pages_mapped" ); goto out; } -if (unlikely (atomic_read (&folio->_pincount))) { -bad_page (page,"nonzero pincount" ); - goto out; +if (IS_ENABLED (CONFIG_64BIT )) { +if (unlikely (atomic_read (&folio->_pincount))) { +bad_page (page,"nonzero pincount" ); + goto out; + } }break ;case 2 :
@@ -970,6 +972,12 @@ static int free_tail_page_prepare(struct page *head_page, struct page *page)bad_page (page,"on deferred list" ); goto out; } +if (!IS_ENABLED (CONFIG_64BIT )) { +if (unlikely (atomic_read (&folio->_pincount))) { +bad_page (page,"nonzero pincount" ); + goto out; + } + }break ;case 3 :/* the third tail page: hugetlb specifics overlap ->mappings */
Signature · 1 line
2.48.1
PATCH v2 6/20 mm: move _entire_mapcount in folio to page[2] on 32bit
In reply to David HildenbrandLet's free up some space on 32bit in page[1] by moving the _pincount to page[2].
Ordinary folios only use the entire mapcount with PMD mappings, so order-1 folios don't apply. Similarly, hugetlb folios are always larger than order-1, turning the entire mapcount essentially unused for all order-1 folios. Moving it to order-1 folios will not change anything.
On 32bit, simply check in folio_entire_mapcount() whether we have an order-1 folio, and return 0 in that case.
Note that THPs on 32bit are not particularly common (and we don't care too much about performance), but we want to keep it working reliably, because likely we want to use large folios there as well in the future, independent of PMD leaf support.
Once we dynamically allocate "struct folio", the 32bit specifics will go away again; even small folios could then have a pincount.
--- include/linux/mm.h | 2 ++ include/linux/mm_types.h | 3 ++- mm/internal.h | 5 +++-- mm/page_alloc.c | 12 ++++++++---- 4 files changed, 15 insertions(+), 7 deletions(-)
include/linux/mm.h +2 -0
@@ -1333,6 +1333,8 @@ static inline int is_vmalloc_or_module_addr(const void *x)static inline int folio_entire_mapcount (const struct folio *folio) {VM_BUG_ON_FOLIO (!folio_test_large (folio), folio); +if (!IS_ENABLED (CONFIG_64BIT ) &&unlikely (folio_large_order (folio) ==1 )) +return 0 ;return atomic_read (&folio->_entire_mapcount) +1 ; }
include/linux/mm_types.h +2 -1
@@ -385,9 +385,9 @@ struct folio {union {struct {atomic_t _large_mapcount; -atomic_t _entire_mapcount;atomic_t _nr_pages_mapped;#ifdef CONFIG_64BIT +atomic_t _entire_mapcount;atomic_t _pincount;#endif /* CONFIG_64BIT */ };
@@ -409,6 +409,7 @@ struct folio {/* public: */ struct list_head _deferred_list;#ifndef CONFIG_64BIT +atomic_t _entire_mapcount;atomic_t _pincount;#endif /* !CONFIG_64BIT */ /* private: the union with struct page is transitional */
mm/internal.h +3 -2
@@ -721,10 +721,11 @@ static inline void prep_compound_head(struct page *page, unsigned int order)folio_set_order (folio, order);atomic_set (&folio->_large_mapcount,-1 ); -atomic_set (&folio->_entire_mapcount,-1 );atomic_set (&folio->_nr_pages_mapped,0 ); -if (IS_ENABLED (CONFIG_64BIT ) || order >1 ) +if (IS_ENABLED (CONFIG_64BIT ) || order >1 ) {atomic_set (&folio->_pincount,0 ); +atomic_set (&folio->_entire_mapcount,-1 ); + }if (order >1 )INIT_LIST_HEAD (&folio->_deferred_list); }
mm/page_alloc.c +8 -4
@@ -947,10 +947,6 @@ static int free_tail_page_prepare(struct page *head_page, struct page *page)switch (page - head_page) {case 1 :/* the first tail page: these may be in place of ->mapping */ -if (unlikely (folio_entire_mapcount (folio))) { -bad_page (page,"nonzero entire_mapcount" ); - goto out; - }if (unlikely (folio_large_mapcount (folio))) {bad_page (page,"nonzero large_mapcount" ); goto out;
@@ -960,6 +956,10 @@ static int free_tail_page_prepare(struct page *head_page, struct page *page) goto out; }if (IS_ENABLED (CONFIG_64BIT )) { +if (unlikely (atomic_read (&folio->_entire_mapcount) +1 )) { +bad_page (page,"nonzero entire_mapcount" ); + goto out; + }if (unlikely (atomic_read (&folio->_pincount))) {bad_page (page,"nonzero pincount" ); goto out;
@@ -973,6 +973,10 @@ static int free_tail_page_prepare(struct page *head_page, struct page *page) goto out; }if (!IS_ENABLED (CONFIG_64BIT )) { +if (unlikely (atomic_read (&folio->_entire_mapcount) +1 )) { +bad_page (page,"nonzero entire_mapcount" ); + goto out; + }if (unlikely (atomic_read (&folio->_pincount))) {bad_page (page,"nonzero pincount" ); goto out;
Signature · 1 line
2.48.1
PATCH v2 7/20 mm/rmap: pass dst_vma to folio_dup_file_rmap_pte() and friends
In reply to David HildenbrandWe'll need access to the destination MM when modifying the large mapcount of a non-hugetlb large folios next. So pass in the destination VMA.
--- include/linux/rmap.h | 42 +++++++++++++++++++++++++----------------- mm/huge_memory.c | 2 +- mm/memory.c | 10 +++++----- 3 files changed, 31 insertions(+), 23 deletions(-)
include/linux/rmap.h +25 -17
@@ -335,7 +335,8 @@ static inline void hugetlb_remove_rmap(struct folio *folio) } static __always_inline void __folio_dup_file_rmap(struct folio *folio, -struct page *page,int nr_pages,enum rmap_level level) +struct page *page,int nr_pages,struct vm_area_struct *dst_vma, +enum rmap_level level) {const int orig_nr_pages = nr_pages;
@@ -366,45 +367,47 @@ static __always_inline void __folio_dup_file_rmap(struct folio *folio,* @folio: The folio to duplicate the mappings of * @page: The first page to duplicate the mappings of * @nr_pages: The number of pages of which the mapping will be duplicated +* @dst_vma: The destination vm area * * The page range of the folio is defined by [page, page + nr_pages) * * The caller needs to hold the page table lock. */ static inline void folio_dup_file_rmap_ptes (struct folio *folio, -struct page *page,int nr_pages) +struct page *page,int nr_pages,struct vm_area_struct *dst_vma) { -__folio_dup_file_rmap (folio, page, nr_pages,RMAP_LEVEL_PTE ); +__folio_dup_file_rmap (folio, page, nr_pages, dst_vma,RMAP_LEVEL_PTE ); } static __always_inline void folio_dup_file_rmap_pte(struct folio *folio, -struct page *page) +struct page *page,struct vm_area_struct *dst_vma) { -__folio_dup_file_rmap (folio, page,1 ,RMAP_LEVEL_PTE ); +__folio_dup_file_rmap (folio, page,1 , dst_vma,RMAP_LEVEL_PTE ); }/** * folio_dup_file_rmap_pmd - duplicate a PMD mapping of a page range of a folio * @folio: The folio to duplicate the mapping of * @page: The first page to duplicate the mapping of +* @dst_vma: The destination vm area * * The page range of the folio is defined by [page, page + HPAGE_PMD_NR) * * The caller needs to hold the page table lock. */ static inline void folio_dup_file_rmap_pmd (struct folio *folio, -struct page *page) +struct page *page,struct vm_area_struct *dst_vma) {#ifdef CONFIG_TRANSPARENT_HUGEPAGE -__folio_dup_file_rmap (folio, page,HPAGE_PMD_NR ,RMAP_LEVEL_PTE ); +__folio_dup_file_rmap (folio, page,HPAGE_PMD_NR , dst_vma,RMAP_LEVEL_PTE );#else WARN_ON_ONCE (true);#endif } static __always_inline int __folio_try_dup_anon_rmap(struct folio *folio, -struct page *page,int nr_pages,struct vm_area_struct *src_vma, -enum rmap_level level) +struct page *page,int nr_pages,struct vm_area_struct *dst_vma, +struct vm_area_struct *src_vma,enum rmap_level level) {const int orig_nr_pages = nr_pages;bool maybe_pinned;
@@ -470,6 +473,7 @@ static __always_inline int __folio_try_dup_anon_rmap(struct folio *folio,* @folio: The folio to duplicate the mappings of * @page: The first page to duplicate the mappings of * @nr_pages: The number of pages of which the mapping will be duplicated +* @dst_vma: The destination vm area * @src_vma: The vm area from which the mappings are duplicated * * The page range of the folio is defined by [page, page + nr_pages)
@@ -488,16 +492,18 @@ static __always_inline int __folio_try_dup_anon_rmap(struct folio *folio,* Returns 0 if duplicating the mappings succeeded. Returns -EBUSY otherwise. */ static inline int folio_try_dup_anon_rmap_ptes (struct folio *folio, -struct page *page,int nr_pages,struct vm_area_struct *src_vma) +struct page *page,int nr_pages,struct vm_area_struct *dst_vma, +struct vm_area_struct *src_vma) { -return __folio_try_dup_anon_rmap (folio, page, nr_pages, src_vma, -RMAP_LEVEL_PTE ); +return __folio_try_dup_anon_rmap (folio, page, nr_pages, dst_vma, + src_vma,RMAP_LEVEL_PTE ); } static __always_inline int folio_try_dup_anon_rmap_pte(struct folio *folio, -struct page *page,struct vm_area_struct *src_vma) +struct page *page,struct vm_area_struct *dst_vma, +struct vm_area_struct *src_vma) { -return __folio_try_dup_anon_rmap (folio, page,1 , src_vma, +return __folio_try_dup_anon_rmap (folio, page,1 , dst_vma, src_vma,RMAP_LEVEL_PTE ); }
@@ -506,6 +512,7 @@ static __always_inline int folio_try_dup_anon_rmap_pte(struct folio *folio,* of a folio * @folio: The folio to duplicate the mapping of * @page: The first page to duplicate the mapping of +* @dst_vma: The destination vm area * @src_vma: The vm area from which the mapping is duplicated * * The page range of the folio is defined by [page, page + HPAGE_PMD_NR)
@@ -524,11 +531,12 @@ static __always_inline int folio_try_dup_anon_rmap_pte(struct folio *folio,* Returns 0 if duplicating the mapping succeeded. Returns -EBUSY otherwise. */ static inline int folio_try_dup_anon_rmap_pmd (struct folio *folio, -struct page *page,struct vm_area_struct *src_vma) +struct page *page,struct vm_area_struct *dst_vma, +struct vm_area_struct *src_vma) {#ifdef CONFIG_TRANSPARENT_HUGEPAGE -return __folio_try_dup_anon_rmap (folio, page,HPAGE_PMD_NR , src_vma, -RMAP_LEVEL_PMD ); +return __folio_try_dup_anon_rmap (folio, page,HPAGE_PMD_NR , dst_vma, + src_vma,RMAP_LEVEL_PMD );#else WARN_ON_ONCE (true);return -EBUSY ;
mm/huge_memory.c +1 -1
@@ -1782,7 +1782,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, src_folio =page_folio (src_page);folio_get (src_folio); -if (unlikely (folio_try_dup_anon_rmap_pmd (src_folio, src_page, src_vma))) { +if (unlikely (folio_try_dup_anon_rmap_pmd (src_folio, src_page, dst_vma, src_vma))) {/* Page maybe pinned: split and retry the fault on PTEs. */ folio_put (src_folio);pte_free (dst_mm, pgtable);
mm/memory.c +5 -5
@@ -845,7 +845,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,folio_get (folio); rss[mm_counter (folio)]++;/* Cannot fail as these pages cannot get pinned. */ -folio_try_dup_anon_rmap_pte (folio, page, src_vma); +folio_try_dup_anon_rmap_pte (folio, page, dst_vma, src_vma);/* * We do not preserve soft-dirty information, because so
@@ -999,14 +999,14 @@ copy_present_ptes(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vmafolio_ref_add (folio, nr);if (folio_test_anon (folio)) {if (unlikely (folio_try_dup_anon_rmap_ptes (folio, page, - nr, src_vma))) { + nr, dst_vma, src_vma))) {folio_ref_sub (folio, nr);return -EAGAIN ; } rss[MM_ANONPAGES ] += nr;VM_WARN_ON_FOLIO (PageAnonExclusive (page), folio); }else { -folio_dup_file_rmap_ptes (folio, page, nr); +folio_dup_file_rmap_ptes (folio, page, nr, dst_vma); rss[mm_counter_file (folio)] += nr; }if (any_writable)
@@ -1024,7 +1024,7 @@ copy_present_ptes(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma* guarantee the pinned page won't be randomly replaced in the * future. */ -if (unlikely (folio_try_dup_anon_rmap_pte (folio, page, src_vma))) { +if (unlikely (folio_try_dup_anon_rmap_pte (folio, page, dst_vma, src_vma))) {/* Page may be pinned, we have to copy. */ folio_put (folio); err = copy_present_page(dst_vma, src_vma, dst_pte, src_pte,
@@ -1034,7 +1034,7 @@ copy_present_ptes(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma rss[MM_ANONPAGES ]++;VM_WARN_ON_FOLIO (PageAnonExclusive (page), folio); }else { -folio_dup_file_rmap_pte (folio, page); +folio_dup_file_rmap_pte (folio, page, dst_vma); rss[mm_counter_file (folio)]++; }
Signature · 1 line
2.48.1
PATCH v2 8/20 mm/rmap: pass vma to __folio_add_rmap()
In reply to David HildenbrandWe'll need access to the destination MM when modifying the mapcount large folios next. So pass in the VMA.
--- mm/rmap.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
mm/rmap.c +4 -4
@@ -1242,8 +1242,8 @@ int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages, pgoff_t pgoff, } static __always_inline unsigned int __folio_add_rmap(struct folio *folio, -struct page *page,int nr_pages,enum rmap_level level, -int *nr_pmdmapped) +struct page *page,int nr_pages,struct vm_area_struct *vma, +enum rmap_level level,int *nr_pmdmapped) {atomic_t *mapped = &folio->_nr_pages_mapped;const int orig_nr_pages = nr_pages;
@@ -1411,7 +1411,7 @@ static __always_inline void __folio_add_anon_rmap(struct folio *folio,VM_WARN_ON_FOLIO (!folio_test_anon (folio), folio); - nr =__folio_add_rmap (folio, page, nr_pages, level, &nr_pmdmapped); + nr =__folio_add_rmap (folio, page, nr_pages, vma, level, &nr_pmdmapped);if (likely (!folio_test_ksm (folio)))__page_check_anon_rmap (folio, page, vma, address);
@@ -1582,7 +1582,7 @@ static __always_inline void __folio_add_file_rmap(struct folio *folio,VM_WARN_ON_FOLIO (folio_test_anon (folio), folio); - nr =__folio_add_rmap (folio, page, nr_pages, level, &nr_pmdmapped); + nr =__folio_add_rmap (folio, page, nr_pages, vma, level, &nr_pmdmapped);__folio_mod_stat (folio, nr, nr_pmdmapped);/* See comments in folio_add_anon_rmap_*() */
Signature · 1 line
2.48.1
PATCH v2 9/20 mm/rmap: abstract large mapcount operations for large folios (!hugetlb)
In reply to David HildenbrandLet's abstract the operations so we can extend these operations easily.
--- include/linux/rmap.h | 32 ++++++++++++++++++++++++++++---- mm/rmap.c | 14 ++++++-------- 2 files changed, 34 insertions(+), 12 deletions(-)
include/linux/rmap.h +28 -4
@@ -173,6 +173,30 @@ static inline void anon_vma_merge(struct vm_area_struct *vma,struct anon_vma *folio_get_anon_vma (const struct folio *folio); +static inline void folio_set_large_mapcount (struct folio *folio,int mapcount, +struct vm_area_struct *vma) +{ +/* Note: mapcounts start at -1. */ +atomic_set (&folio->_large_mapcount, mapcount -1 ); +} + +static inline void folio_add_large_mapcount (struct folio *folio, +int diff,struct vm_area_struct *vma) +{ +atomic_add (diff, &folio->_large_mapcount); +} + +static inline void folio_sub_large_mapcount (struct folio *folio, +int diff,struct vm_area_struct *vma) +{ +atomic_sub (diff, &folio->_large_mapcount); +} + +#define folio_inc_large_mapcount (folio, vma) \ + folio_add_large_mapcount(folio, 1, vma) +#define folio_dec_large_mapcount (folio, vma) \ + folio_sub_large_mapcount(folio, 1, vma) +/* RMAP flags, currently only relevant for some anon rmap operations. */ typedef int __bitwise rmap_t ;
@@ -352,12 +376,12 @@ static __always_inline void __folio_dup_file_rmap(struct folio *folio,do {atomic_inc (&page->_mapcount); }while (page++, --nr_pages >0 ); -atomic_add (orig_nr_pages, &folio->_large_mapcount); +folio_add_large_mapcount (folio, orig_nr_pages, dst_vma);break ;case RMAP_LEVEL_PMD :case RMAP_LEVEL_PUD :atomic_inc (&folio->_entire_mapcount); -atomic_inc (&folio->_large_mapcount); +folio_inc_large_mapcount (folio, dst_vma);break ; } }
@@ -451,7 +475,7 @@ static __always_inline int __folio_try_dup_anon_rmap(struct folio *folio,ClearPageAnonExclusive (page);atomic_inc (&page->_mapcount); }while (page++, --nr_pages >0 ); -atomic_add (orig_nr_pages, &folio->_large_mapcount); +folio_add_large_mapcount (folio, orig_nr_pages, dst_vma);break ;case RMAP_LEVEL_PMD :case RMAP_LEVEL_PUD :
@@ -461,7 +485,7 @@ static __always_inline int __folio_try_dup_anon_rmap(struct folio *folio,ClearPageAnonExclusive (page); }atomic_inc (&folio->_entire_mapcount); -atomic_inc (&folio->_large_mapcount); +folio_inc_large_mapcount (folio, dst_vma);break ; }return 0 ;
mm/rmap.c +6 -8
@@ -1266,7 +1266,7 @@ static __always_inline unsigned int __folio_add_rmap(struct folio *folio, atomic_add_return_relaxed(first, mapped) < ENTIRELY_MAPPED) nr = first; -atomic_add (orig_nr_pages, &folio->_large_mapcount); +folio_add_large_mapcount (folio, orig_nr_pages, vma);break ;case RMAP_LEVEL_PMD :case RMAP_LEVEL_PUD :
@@ -1290,7 +1290,7 @@ static __always_inline unsigned int __folio_add_rmap(struct folio *folio, nr =0 ; } } -atomic_inc (&folio->_large_mapcount); +folio_inc_large_mapcount (folio, vma);break ; }return nr;
@@ -1556,14 +1556,12 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma,SetPageAnonExclusive (page); } -/* increment count (starts at -1) */ -atomic_set (&folio->_large_mapcount, nr -1 ); +folio_set_large_mapcount (folio, nr, vma);atomic_set (&folio->_nr_pages_mapped, nr); }else {/* increment count (starts at -1) */ atomic_set (&folio->_entire_mapcount,0 ); -/* increment count (starts at -1) */ -atomic_set (&folio->_large_mapcount,0 ); +folio_set_large_mapcount (folio,1 , vma);atomic_set (&folio->_nr_pages_mapped,ENTIRELY_MAPPED );if (exclusive)SetPageAnonExclusive (&folio->page);
@@ -1665,7 +1663,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,break ; } -atomic_sub (nr_pages, &folio->_large_mapcount); +folio_sub_large_mapcount (folio, nr_pages, vma);do { last +=atomic_add_negative (-1 , &page->_mapcount); }while (page++, --nr_pages >0 );
@@ -1678,7 +1676,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,break ;case RMAP_LEVEL_PMD :case RMAP_LEVEL_PUD : -atomic_dec (&folio->_large_mapcount); +folio_dec_large_mapcount (folio, vma); last =atomic_add_negative (-1 , &folio->_entire_mapcount);if (last) { nr =atomic_sub_return_relaxed (ENTIRELY_MAPPED , mapped);
Signature · 1 line
2.48.1
PATCH v2 10/20 bit_spinlock: __always_inline (un)lock functions
In reply to David HildenbrandThe compiler might decide that it is a smart idea to not inline bit_spin_lock(), primarily when a couple of functions in the same file end up calling it. Especially when used in RMAP map/unmap code next, the compiler sometimes decides to not inline, which is then observable in some micro-benchmarks.
Let's simply flag all lock/unlock functions as __always_inline; arch_test_and_set_bit_lock() and friends are already tagged like that (but not test_and_set_bit_lock() for some reason).
If ever a problem, we could split it into a fast and a slow path, and only force the fast path to be inlined. But there is nothing particularly "big" here.
--- include/linux/bit_spinlock.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
include/linux/bit_spinlock.h +4 -4
@@ -13,7 +13,7 @@* Don't use this unless you really need to: spin_lock() and spin_unlock() * are significantly faster. */ -static inline void bit_spin_lock (int bitnum,unsigned long *addr) +static __always_inline void bit_spin_lock(int bitnum, unsigned long *addr) {/* * Assuming the lock is uncontended, this never enters
@@ -38,7 +38,7 @@ static inline void bit_spin_lock(int bitnum, unsigned long *addr)/* * Return true if it was acquired */ -static inline int bit_spin_trylock (int bitnum,unsigned long *addr) +static __always_inline int bit_spin_trylock(int bitnum, unsigned long *addr) {preempt_disable ();#if defined(CONFIG_SMP ) || defined(CONFIG_DEBUG_SPINLOCK )
@@ -54,7 +54,7 @@ static inline int bit_spin_trylock(int bitnum, unsigned long *addr)/* * bit-based spin_unlock() */ -static inline void bit_spin_unlock (int bitnum,unsigned long *addr) +static __always_inline void bit_spin_unlock(int bitnum, unsigned long *addr) {#ifdef CONFIG_DEBUG_SPINLOCK BUG_ON (!test_bit (bitnum, addr));
@@ -71,7 +71,7 @@ static inline void bit_spin_unlock(int bitnum, unsigned long *addr)* non-atomic version, which can be used eg. if the bit lock itself is * protecting the rest of the flags in the word. */ -static inline void __bit_spin_unlock (int bitnum,unsigned long *addr) +static __always_inline void __bit_spin_unlock(int bitnum, unsigned long *addr) {#ifdef CONFIG_DEBUG_SPINLOCK BUG_ON (!test_bit (bitnum, addr));
Signature · 1 line
2.48.1
PATCH v2 11/20 mm/rmap: use folio_large_nr_pages() in add/remove functions
In reply to David HildenbrandLet's just use the "large" variant in code where we are sure that we have a large folio in our hands: this way we are sure that we don't perform any unnecessary "large" checks.
While at it, convert the VM_BUG_ON_VMA to a VM_WARN_ON_ONCE.
Maybe in the future there will not be a difference in that regard between large and small folios; in that case, unifying the handling again will be easy. E.g., folio_large_nr_pages() will simply translate to folio_nr_pages() until we replace all instances.
--- mm/rmap.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-)
mm/rmap.c +8 -6
@@ -1274,7 +1274,7 @@ static __always_inline unsigned int __folio_add_rmap(struct folio *folio,if (first) { nr =atomic_add_return_relaxed (ENTIRELY_MAPPED , mapped);if (likely (nr <ENTIRELY_MAPPED +ENTIRELY_MAPPED )) { - nr_pages =folio_nr_pages (folio); + nr_pages =folio_large_nr_pages (folio);/* * We only track PMD mappings of PMD-sized * folios separately.
@@ -1522,14 +1522,11 @@ void folio_add_anon_rmap_pmd(struct folio *folio, struct page *page,void folio_add_new_anon_rmap (struct folio *folio,struct vm_area_struct *vma,unsigned long address,rmap_t flags) { -const int nr =folio_nr_pages (folio);const bool exclusive = flags &RMAP_EXCLUSIVE ; -int nr_pmdmapped =0 ; +int nr =1 , nr_pmdmapped =0 ;VM_WARN_ON_FOLIO (folio_test_hugetlb (folio), folio);VM_WARN_ON_FOLIO (!exclusive && !folio_test_locked (folio), folio); -VM_BUG_ON_VMA (address < vma->vm_start || - address + (nr <<PAGE_SHIFT ) > vma->vm_end, vma);/* * VM_DROPPABLE mappings don't swap; instead they're just dropped when
@@ -1547,6 +1544,7 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, }else if (!folio_test_pmd_mappable (folio)) {int i; + nr =folio_large_nr_pages (folio);for (i =0 ; i < nr; i++) {struct page *page =folio_page (folio, i);
@@ -1559,6 +1557,7 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma,folio_set_large_mapcount (folio, nr, vma);atomic_set (&folio->_nr_pages_mapped, nr); }else { + nr =folio_large_nr_pages (folio);/* increment count (starts at -1) */ atomic_set (&folio->_entire_mapcount,0 );folio_set_large_mapcount (folio,1 , vma);
@@ -1568,6 +1567,9 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, nr_pmdmapped = nr; } +VM_WARN_ON_ONCE (address < vma->vm_start || + address + (nr <<PAGE_SHIFT ) > vma->vm_end); +__folio_mod_stat (folio, nr, nr_pmdmapped);mod_mthp_stat (folio_order (folio),MTHP_STAT_NR_ANON ,1 ); }
@@ -1681,7 +1683,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,if (last) { nr =atomic_sub_return_relaxed (ENTIRELY_MAPPED , mapped);if (likely (nr <ENTIRELY_MAPPED )) { - nr_pages =folio_nr_pages (folio); + nr_pages =folio_large_nr_pages (folio);if (level ==RMAP_LEVEL_PMD ) nr_pmdmapped = nr_pages; nr = nr_pages - (nr &FOLIO_PAGES_MAPPED );
Signature · 1 line
2.48.1
PATCH v2 12/20 mm/rmap: basic MM owner tracking for large folios (!hugetlb)
In reply to David HildenbrandFor small folios, we traditionally use the mapcount to decide whether it was "certainly mapped exclusively" by a single MM (mapcount == 1) or whether it "maybe mapped shared" by multiple MMs (mapcount > 1). For PMD-sized folios that were PMD-mapped, we were able to use a similar mechanism (single PMD mapping), but for PTE-mapped folios and in the future folios that span multiple PMDs, this does not work.
So we need a different mechanism to handle large folios. Let's add a new mechanism to detect whether a large folio is "certainly mapped exclusively", or whether it is "maybe mapped shared".
We'll use this information next to optimize CoW reuse for PTE-mapped anonymous THP, and to convert folio_likely_mapped_shared() to folio_maybe_mapped_shared(), independent of per-page mapcounts.
For each large folio, we'll have two slots, whereby a slot stores:
(1) an MM id: unique id assigned to each MM
(2) a per-MM mapcount
If a slot is unoccupied, it can be taken by the next MM that maps folio page.
In addition, we'll remember the current state -- "mapped exclusively" vs. "maybe mapped shared" -- and use a bit spinlock to sync on updates and to reduce the total number of atomic accesses on updates. In the future, it might be possible to squeeze a proper spinlock into "struct folio". For now, keep it simple, as we require the whole thing with THP only, that is incompatible with RT.
As we have to squeeze this information into the "struct folio" of even folios of order-1 (2 pages), and we generally want to reduce the required metadata, we'll assign each MM a unique ID that can fit into an int. In total, we can squeeze everything into 4x int (2x long) on 64bit.
32bit support is a bit challenging, because we only have 2x long == 2x int in order-1 folios. But we can make it work for now, because we neither expect many MMs nor very large folios on 32bit.
We will reliably detect folios as "mapped exclusively" vs. "mapped shared" as long as only two MMs map pages of a folio at one point in time -- for example with fork() and short-lived child processes, or with apps that hand over state from one instance to another.
As soon as three MMs are involved at the same time, we might detect "maybe mapped shared" although the folio is "mapped exclusively".
Example 1:
(1) App1 faults in a (shmem/file-backed) folio page -> Tracked as MM0
(2) App2 faults in a folio page -> Tracked as MM1
(4) App1 unmaps all folio pages
-> We will detect "mapped exclusively".
Example 2:
(1) App1 faults in a (shmem/file-backed) folio page -> Tracked as MM0
(2) App2 faults in a folio page -> Tracked as MM1
(3) App3 faults in a folio page -> No slot available, tracked as "unknown"
(4) App1 and App2 unmap all folio pages
-> We will detect "maybe mapped shared".
Make use of __always_inline to keep possible performance degradation when (un)mapping large folios to a minimum.
Note: by squeezing the two flags into the "unsigned long" that stores the MM ids, we can use non-atomic __bit_spin_unlock() and non-atomic setting/clearing of the "maybe mapped shared" bit, effectively not adding any new atomics on the hot path when updating the large mapcount + new metadata, which further helps reduce the runtime overhead in micro-benchmarks.
--- Documentation/mm/transhuge.rst | 8 ++ include/linux/mm_types.h | 49 ++++++++++ include/linux/page-flags.h | 4 + include/linux/rmap.h | 165 +++++++++++++++++++++++++++++++++ kernel/fork.c | 36 +++++++ mm/Kconfig | 4 + mm/internal.h | 5 + mm/page_alloc.c | 10 ++ 8 files changed, 281 insertions(+)
Documentation/mm/transhuge.rst +8 -0
@@ -120,11 +120,19 @@ pages: and also increment/decrement folio->_nr_pages_mapped by ENTIRELY_MAPPED when _entire_mapcount goes from -1 to 0 or 0 to -1. + We also maintain the two slots for tracking MM owners (MM ID and + corresponding mapcount), and the current status ("maybe mapped shared" vs. + "mapped exclusively"). + - map/unmap of individual pages with PTE entry increment/decrement page->_mapcount, increment/decrement folio->_large_mapcount and also increment/decrement folio->_nr_pages_mapped when page->_mapcount goes from -1 to 0 or 0 to -1 as this counts the number of pages mapped by PTE. + We also maintain the two slots for tracking MM owners (MM ID and + corresponding mapcount), and the current status ("maybe mapped shared" vs. + "mapped exclusively"). + split_huge_page internally has to distribute the refcounts in the head page to the tail pages before clearing all PG_head/tail bits from the page structures. It can be done easily for refcounts taken by page table
include/linux/mm_types.h +49 -0
73 lines not on this page · raw message
include/linux/page-flags.h +4 -0
@@ -1222,6 +1222,10 @@ static inline int folio_has_private(const struct folio *folio)return !!(folio->flags &PAGE_FLAGS_PRIVATE ); } +static inline bool folio_test_large_maybe_mapped_shared (const struct folio *folio) +{ +return test_bit (FOLIO_MM_IDS_SHARED_BITNUM , &folio->_mm_ids); +}#undef PF_ANY#undef PF_HEAD#undef PF_NO_TAIL
include/linux/rmap.h +165 -0
183 lines not on this page · raw message
kernel/fork.c +36 -0
@@ -802,6 +802,36 @@ static int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm)#define mm_free_pgd (mm)#endif /* CONFIG_MMU */ +#ifdef CONFIG_MM_ID +static DEFINE_IDA (mm_ida); + +static inline int mm_alloc_id (struct mm_struct *mm) +{ +int ret; + + ret =ida_alloc_range (&mm_ida,MM_ID_MIN ,MM_ID_MAX ,GFP_KERNEL ); +if (ret <0 ) +return ret; + mm->mm_id = ret; +return 0 ; +} + +static inline void mm_free_id (struct mm_struct *mm) +{ +const mm_id_t id = mm->mm_id; + + mm->mm_id =MM_ID_DUMMY ; +if (id ==MM_ID_DUMMY ) +return ; +if (WARN_ON_ONCE (id <MM_ID_MIN || id >MM_ID_MAX )) +return ; +ida_free (&mm_ida, id); +} +#else /* !CONFIG_MM_ID */ +static inline int mm_alloc_id (struct mm_struct *mm) {return 0 ; } +static inline void mm_free_id (struct mm_struct *mm) {} +#endif /* CONFIG_MM_ID */ +static void check_mm (struct mm_struct *mm) {int i;
@@ -905,6 +935,7 @@ void __mmdrop(struct mm_struct *mm)WARN_ON_ONCE (mm == current->active_mm);mm_free_pgd (mm); +mm_free_id (mm);destroy_context (mm);mmu_notifier_subscriptions_destroy (mm);check_mm (mm);
@@ -1289,6 +1320,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p,if (mm_alloc_pgd (mm)) goto fail_nopgd; +if (mm_alloc_id (mm)) + goto fail_noid; +if (init_new_context (p, mm)) goto fail_nocontext;
@@ -1308,6 +1342,8 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, fail_cid:destroy_context (mm); fail_nocontext: +mm_free_id (mm); +fail_noid:mm_free_pgd (mm); fail_nopgd:free_mm (mm);
mm/Kconfig +4 -0
@@ -817,11 +817,15 @@ config ARCH_WANT_GENERAL_HUGETLB config ARCH_WANTS_THP_SWAP def_bool n +config MM_ID + def_bool n + menuconfig TRANSPARENT_HUGEPAGE bool "Transparent Hugepage Support" depends on HAVE_ARCH_TRANSPARENT_HUGEPAGE && !PREEMPT_RT select COMPACTION select XARRAY_MULTI + select MM_ID help Transparent Hugepages allows the kernel to use huge pages and huge tlb transparently to the applications whenever possible.
mm/internal.h +5 -0
@@ -722,6 +722,11 @@ static inline void prep_compound_head(struct page *page, unsigned int order)folio_set_order (folio, order);atomic_set (&folio->_large_mapcount,-1 );atomic_set (&folio->_nr_pages_mapped,0 ); +if (IS_ENABLED (CONFIG_MM_ID )) { + folio->_mm_ids =0 ; + folio->_mm_id_mapcount[0 ] =-1 ; + folio->_mm_id_mapcount[1 ] =-1 ; + }if (IS_ENABLED (CONFIG_64BIT ) || order >1 ) {atomic_set (&folio->_pincount,0 );atomic_set (&folio->_entire_mapcount,-1 );
mm/page_alloc.c +10 -0
@@ -955,6 +955,16 @@ static int free_tail_page_prepare(struct page *head_page, struct page *page)bad_page (page,"nonzero nr_pages_mapped" ); goto out; } +if (IS_ENABLED (CONFIG_MM_ID )) { +if (unlikely (folio->_mm_id_mapcount[0 ] !=-1 )) { +bad_page (page,"nonzero mm mapcount 0" ); + goto out; + } +if (unlikely (folio->_mm_id_mapcount[1 ] !=-1 )) { +bad_page (page,"nonzero mm mapcount 1" ); + goto out; + } + }if (IS_ENABLED (CONFIG_64BIT )) {if (unlikely (atomic_read (&folio->_entire_mapcount) +1 )) {bad_page (page,"nonzero entire_mapcount" );
Signature · 1 line
2.48.1
PATCH v2 13/20 mm: Copy-on-Write (COW) reuse support for PTE-mapped THP
In reply to David HildenbrandCurrently, we never end up reusing PTE-mapped THPs after fork. This wasn't really a problem with PMD-sized THPs, because they would have to be PTE-mapped first, but it's getting a problem with smaller THP sizes that are effectively always PTE-mapped.
With our new "mapped exclusively" vs "maybe mapped shared" logic for large folios, implementing CoW reuse for PTE-mapped THPs is straight
forward: if exclusively mapped, make sure that all references are from these (our) mappings. Add some helpful comments to explain the details.
CONFIG_TRANSPARENT_HUGEPAGE selects CONFIG_MM_ID. If we spot an anon large folio without CONFIG_TRANSPARENT_HUGEPAGE in that code, something is seriously messed up.
There are plenty of things we can optimize in the future: For example, we could remember that the folio is fully exclusive so we could speedup the next fault further. Also, we could try "faulting around", turning surrounding PTEs that map the same folio writable. But especially the latter might increase COW latency, so it would need further investigation.
--- mm/memory.c | 84 ++++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 76 insertions(+), 8 deletions(-)
mm/memory.c +76 -8
@@ -3706,19 +3706,87 @@ static vm_fault_t wp_page_shared(struct vm_fault *vmf, struct folio *folio)return ret; } -static bool wp_can_reuse_anon_folio (struct folio *folio, -struct vm_area_struct *vma) +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +static bool __wp_can_reuse_large_anon_folio (struct folio *folio, +struct vm_area_struct *vma) { +bool exclusive = false; + +/* Let's just free up a large folio if only a single page is mapped. */ +if (folio_large_mapcount (folio) <=1 ) +return false; +/* -* We could currently only reuse a subpage of a large folio if no -* other subpages of the large folios are still mapped. However, -* let's just consistently not reuse subpages even if we could -* reuse in that scenario, and give back a large folio a bit -* sooner. +* The assumption for anonymous folios is that each page can only get +* mapped once into each MM. The only exception are KSM folios, which +* are always small. +* +* Each taken mapcount must be paired with exactly one taken reference, +* whereby the refcount must be incremented before the mapcount when +* mapping a page, and the refcount must be decremented after the +* mapcount when unmapping a page. +* +* If all folio references are from mappings, and all mappings are in +* the page tables of this MM, then this folio is exclusive to this MM. */ -if (folio_test_large (folio)) +if (folio_test_large_maybe_mapped_shared (folio)) +return false; + +VM_WARN_ON_ONCE (folio_test_ksm (folio)); +VM_WARN_ON_ONCE (folio_mapcount (folio) >folio_nr_pages (folio)); +VM_WARN_ON_ONCE (folio_entire_mapcount (folio)); + +if (unlikely (folio_test_swapcache (folio))) { +/* +* Note: freeing up the swapcache will fail if some PTEs are +* still swap entries. +*/ +if (!folio_trylock (folio)) +return false; +folio_free_swap (folio); +folio_unlock (folio); + } + +if (folio_large_mapcount (folio) !=folio_ref_count (folio))return false; +/* Stabilize the mapcount vs. refcount and recheck. */ +folio_lock_large_mapcount (folio); +VM_WARN_ON_ONCE (folio_large_mapcount (folio) <folio_ref_count (folio)); + +if (folio_test_large_maybe_mapped_shared (folio)) + goto unlock; +if (folio_large_mapcount (folio) !=folio_ref_count (folio)) + goto unlock; + +VM_WARN_ON_ONCE (folio_mm_id (folio,0 ) != vma->vm_mm->mm_id && +folio_mm_id (folio,1 ) != vma->vm_mm->mm_id); + +/* +* Do we need the folio lock? Likely not. If there would have been +* references from page migration/swapout, we would have detected +* an additional folio reference and never ended up here. +*/ + exclusive = true; +unlock: +folio_unlock_large_mapcount (folio); +return exclusive; +} +#else /* !CONFIG_TRANSPARENT_HUGEPAGE */ +static bool __wp_can_reuse_large_anon_folio (struct folio *folio, +struct vm_area_struct *vma) +{ +BUILD_BUG (); +} +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ + +static bool wp_can_reuse_anon_folio (struct folio *folio, +struct vm_area_struct *vma) +{ +if (IS_ENABLED (CONFIG_TRANSPARENT_HUGEPAGE ) &&folio_test_large (folio)) +return __wp_can_reuse_large_anon_folio (folio, vma); +VM_WARN_ON_ONCE (folio_test_large (folio)); +/* * We have to verify under folio lock: these early checks are * just an optimization to avoid locking the folio and freeing
Signature · 1 line
2.48.1
PATCH v2 14/20 mm: convert folio_likely_mapped_shared() to folio_maybe_mapped_shared()
In reply to David HildenbrandLet's reuse our new MM ownership tracking infrastructure for large folios to make folio_likely_mapped_shared() never return false negatives -- never indicating "not mapped shared" although the folio *is* mapped shared. With that, we can rename it to folio_maybe_mapped_shared() and get rid of the dependency on the mapcount of the first folio page.
The semantics are now arguably clearer: no mixture of "false negatives" and "false positives", only the remaining possibility for "false positives".
Thoroughly document the new semantics. We might now detect that a large folio is "maybe mapped shared" although it *no longer* is -- but once was. Now, if more than two MMs mapped a folio at the same time, and the MM mapping the folio exclusively at the end is not one tracked in the two folio MM slots, we will detect the folio as "maybe mapped shared".
For anonymous folios, usually (except weird corner cases) all PTEs that target a "maybe mapped shared" folio are R/O. As soon as a child process would write to them (iow, actively use them), we would CoW and effectively replace these PTEs. Most cases (below) are not expected to really matter with large anonymous folios for this reason.
Most importantly, there will be no change at all for:
* small folios
* hugetlb folios
* PMD-mapped PMD-sized THPs (single mapping)
This change has the potential to affect existing callers of folio_likely_mapped_shared() -> folio_maybe_mapped_shared():
(1) fs/proc/task_mmu.c: no change (hugetlb)
(2) khugepaged counts PTEs that target shared folios towards max_ptes_shared (default: HPAGE_PMD_NR / 2), meaning we could skip a collapse where we would have previously collapsed. This only applies to anonymous folios and is not expected to matter in practice.
Worth noting that this change sorts out case (A) documented in commit 1bafe96e89f0 ("mm/khugepaged: replace page_mapcount() check by folio_likely_mapped_shared()") by removing the possibility for "false negatives".
(3) MADV_COLD / MADV_PAGEOUT / MADV_FREE will not try splitting PTE-mapped THPs that are considered shared but not fully covered by the requested range, consequently not processing them.
PMD-mapped PMD-sized THP are not affected, or when all PTEs are covered. These functions are usually only called on anon/file folios that are exclusively mapped most of the time (no other file mappings or no fork()), so the "false negatives" are not expected to matter in practice.
(4) mbind() / migrate_pages() / move_pages() will refuse to migrate shared folios unless MPOL_MF_MOVE_ALL is effective (requires CAP_SYS_NICE). We will now reject some folios that could be migrated.
Similar to (3), especially with MPOL_MF_MOVE_ALL, so this is not expected to matter in practice.
Note that cpuset_migrate_mm_workfn() calls do_migrate_pages() with MPOL_MF_MOVE_ALL.
(5) NUMA hinting
mm/migrate.c:migrate_misplaced_folio_prepare() will skip file folios that are probably shared libraries (-> "mapped shared" and executable). This check would have detected it as a shared library at some point (at least 3 MMs mapping it), so detecting it afterwards does not sound wrong (still a shared library). Not expected to matter.
mm/memory.c:numa_migrate_check() will indicate TNF_SHARED in MAP_SHARED file mappings when encountering a shared folio. Similar reasoning, not expected to matter.
mm/mprotect.c:change_pte_range() will skip folios detected as shared in CoW mappings. Similarly, this is not expected to matter in practice, but if it would ever be a problem we could relax that check a bit (e.g., basing it on the average page-mapcount in a folio), because it was only an optimization when many (e.g., 288) processes were mapping the same folios -- see commit 859d4adc3415 ("mm: numa: do not trap faults on shared data section pages.")
(6) mm/rmap.c:folio_referenced_one() will skip exclusive swapbacked folios in dying processes. Applies to anonymous folios only. Without "false negatives", we'll now skip all actually shared ones. Skipping ones that are actually exclusive won't really matter, it's a pure optimization, and is not expected to matter in practice.
In theory, one can detect the problematic scenario: folio_mapcount() > 0 and no folio MM slot is occupied ("state unknown"). One could reset the MM slots while doing an rmap walk, which migration / folio split already do when setting everything up. Further, when batching PTEs we might naturally learn about a owner (e.g., folio_mapcount() == nr_ptes) and could update the owner. However, we'll defer that until the scenarios where it would really matter are clear.
--- fs/proc/task_mmu.c | 4 ++-- include/linux/mm.h | 43 ++++++++++++++++++++++--------------------- mm/huge_memory.c | 2 +- mm/khugepaged.c | 8 +++----- mm/madvise.c | 6 +++--- mm/memory.c | 2 +- mm/mempolicy.c | 8 ++++---- mm/migrate.c | 7 +++---- mm/mprotect.c | 2 +- mm/rmap.c | 2 +- 10 files changed, 41 insertions(+), 43 deletions(-)
fs/proc/task_mmu.c +2 -2
@@ -1023,7 +1023,7 @@ static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask,if (folio) {/* We treat non-present entries as "maybe shared". */ -if (!present ||folio_likely_mapped_shared (folio) || +if (!present ||folio_maybe_mapped_shared (folio) ||hugetlb_pmd_shared (pte)) mss->shared_hugetlb +=huge_page_size (hstate_vma (vma));else
@@ -1879,7 +1879,7 @@ static int pagemap_hugetlb_range(pte_t *ptep, unsigned long hmask,if (!folio_test_anon (folio)) flags |=PM_FILE ; -if (!folio_likely_mapped_shared (folio) && +if (!folio_maybe_mapped_shared (folio) && !hugetlb_pmd_shared (ptep)) flags |=PM_MMAP_EXCLUSIVE ;
include/linux/mm.h +22 -21
76 lines not on this page · raw message
mm/huge_memory.c +1 -1
@@ -2155,7 +2155,7 @@ bool madvise_free_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,* If other processes are mapping this folio, we couldn't discard * the folio unless they all do MADV_FREE so let's skip the folio. */ -if (folio_likely_mapped_shared (folio)) +if (folio_maybe_mapped_shared (folio)) goto out;if (!folio_trylock (folio))
mm/khugepaged.c +3 -5
@@ -607,7 +607,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,VM_BUG_ON_FOLIO (!folio_test_anon (folio), folio);/* See hpage_collapse_scan_pmd(). */ -if (folio_likely_mapped_shared (folio)) { +if (folio_maybe_mapped_shared (folio)) { ++shared;if (cc->is_khugepaged && shared > khugepaged_max_ptes_shared) {
@@ -1359,11 +1359,9 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,/* * We treat a single page as shared if any part of the THP -* is shared. "False negatives" from -* folio_likely_mapped_shared() are not expected to matter -* much in practice. +* is shared. */ -if (folio_likely_mapped_shared (folio)) { +if (folio_maybe_mapped_shared (folio)) { ++shared;if (cc->is_khugepaged && shared > khugepaged_max_ptes_shared) {
mm/madvise.c +3 -3
@@ -387,7 +387,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, folio =pmd_folio (orig_pmd);/* Do not interfere with other mappings of this folio */ -if (folio_likely_mapped_shared (folio)) +if (folio_maybe_mapped_shared (folio)) goto huge_unlock;if (pageout_anon_only_filter && !folio_test_anon (folio))
@@ -486,7 +486,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,if (nr <folio_nr_pages (folio)) {int err; -if (folio_likely_mapped_shared (folio)) +if (folio_maybe_mapped_shared (folio))continue ;if (pageout_anon_only_filter && !folio_test_anon (folio))continue ;
@@ -721,7 +721,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,if (nr <folio_nr_pages (folio)) {int err; -if (folio_likely_mapped_shared (folio)) +if (folio_maybe_mapped_shared (folio))continue ;if (!folio_trylock (folio))continue ;
mm/memory.c +1 -1
@@ -5672,7 +5672,7 @@ int numa_migrate_check(struct folio *folio, struct vm_fault *vmf,* Flag if the folio is shared between multiple address spaces. This * is later used when determining whether to group tasks together */ -if (folio_likely_mapped_shared (folio) && (vma->vm_flags &VM_SHARED )) +if (folio_maybe_mapped_shared (folio) && (vma->vm_flags &VM_SHARED )) *flags |=TNF_SHARED ;/* * For memory tiering mode, cpupid of slow memory page is used
mm/mempolicy.c +4 -4
@@ -642,11 +642,11 @@ static int queue_folios_hugetlb(pte_t *pte, unsigned long hmask,* Unless MPOL_MF_MOVE_ALL, we try to avoid migrating a shared folio. * Choosing not to migrate a shared folio is not counted as a failure. * -* See folio_likely_mapped_shared() on possible imprecision when we +* See folio_maybe_mapped_shared() on possible imprecision when we * cannot easily detect if a folio is shared. */ if ((flags &MPOL_MF_MOVE_ALL ) || - (!folio_likely_mapped_shared (folio) && !hugetlb_pmd_shared (pte))) + (!folio_maybe_mapped_shared (folio) && !hugetlb_pmd_shared (pte)))if (!folio_isolate_hugetlb (folio, qp->pagelist)) qp->nr_failed++; unlock:
@@ -1033,10 +1033,10 @@ static bool migrate_folio_add(struct folio *folio, struct list_head *foliolist,* Unless MPOL_MF_MOVE_ALL, we try to avoid migrating a shared folio. * Choosing not to migrate a shared folio is not counted as a failure. * -* See folio_likely_mapped_shared() on possible imprecision when we +* See folio_maybe_mapped_shared() on possible imprecision when we * cannot easily detect if a folio is shared. */ -if ((flags &MPOL_MF_MOVE_ALL ) || !folio_likely_mapped_shared (folio)) { +if ((flags &MPOL_MF_MOVE_ALL ) || !folio_maybe_mapped_shared (folio)) {if (folio_isolate_lru (folio)) {list_add_tail (&folio->lru, foliolist); node_stat_mod_folio(folio,
mm/migrate.c +3 -4
@@ -2228,7 +2228,7 @@ static int __add_folio_for_migration(struct folio *folio, int node,if (folio_nid (folio) == node)return 0 ; -if (folio_likely_mapped_shared (folio) && !migrate_all) +if (folio_maybe_mapped_shared (folio) && !migrate_all)return -EACCES ;if (folio_test_hugetlb (folio)) {
@@ -2653,11 +2653,10 @@ int migrate_misplaced_folio_prepare(struct folio *folio,* processes with execute permissions as they are probably * shared libraries. * -* See folio_likely_mapped_shared() on possible imprecision +* See folio_maybe_mapped_shared() on possible imprecision * when we cannot easily detect if a folio is shared. */ -if ((vma->vm_flags &VM_EXEC ) && -folio_likely_mapped_shared (folio)) +if ((vma->vm_flags &VM_EXEC ) &&folio_maybe_mapped_shared (folio))return -EACCES ;/*
mm/mprotect.c +1 -1
@@ -133,7 +133,7 @@ static long change_pte_range(struct mmu_gather *tlb,/* Also skip shared copy-on-write pages */ if (is_cow_mapping (vma->vm_flags) && (folio_maybe_dma_pinned (folio) || -folio_likely_mapped_shared (folio))) +folio_maybe_mapped_shared (folio)))continue ;/*
mm/rmap.c +1 -1
@@ -889,7 +889,7 @@ static bool folio_referenced_one(struct folio *folio,if ((!atomic_read (&vma->vm_mm->mm_users) ||check_stable_address_space (vma->vm_mm)) &&folio_test_anon (folio) &&folio_test_swapbacked (folio) && - !folio_likely_mapped_shared (folio)) { + !folio_maybe_mapped_shared (folio)) { pra->referenced =-1 ;page_vma_mapped_walk_done (&pvmw);return false;
Signature · 1 line
2.48.1
PATCH v2 15/20 mm: CONFIG_NO_PAGE_MAPCOUNT to prepare for not maintain per-page mapcounts in large folios
In reply to David HildenbrandWe're close to the finishing line: let's introduce a new CONFIG_NO_PAGE_MAPCOUNT config option where we will incrementally remove any dependencies on per-page mapcounts in large folios. Once that's done, we'll stop maintaining the per-page mapcounts with this config option enabled.
CONFIG_NO_PAGE_MAPCOUNT will be EXPERIMENTAL for now, as we'll have to learn about some of the real world impact of some of the implications.
As writing "!CONFIG_NO_PAGE_MAPCOUNT" is really nasty, let's introduce a helper config option "CONFIG_PAGE_MAPCOUNT" that expresses the negation.
--- mm/Kconfig | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
mm/Kconfig +18 -0
@@ -889,8 +889,26 @@ config READ_ONLY_THP_FOR_FS support of file THPs will be developed in the next few release cycles. +config NO_PAGE_MAPCOUNT + bool "No per-page mapcount (EXPERIMENTAL)" + help + Do not maintain per-page mapcounts for pages part of larger + allocations, such as transparent huge pages. + + When this config option is enabled, some interfaces that relied on + this information will rely on less-precise per-allocation information + instead: for example, using the average per-page mapcount in such + a large allocation instead of the per-page mapcount. + + EXPERIMENTAL because the severity of some of the implications first + have to be understood properly. + endif # TRANSPARENT_HUGEPAGE +# simple helper to make the code a bit easier to read +config PAGE_MAPCOUNT + def_bool !NO_PAGE_MAPCOUNT + # # The architecture supports pgtable leaves that is larger than PAGE_SIZE #
Signature · 1 line
2.48.1
PATCH v2 16/20 fs/proc/page: remove per-page mapcount dependency for /proc/kpagecount (CONFIG_NO_PAGE_MAPCOUNT)
In reply to David HildenbrandLet's implement an alternative when per-page mapcounts in large folios are no longer maintained -- soon with CONFIG_NO_PAGE_MAPCOUNT.
For large folios, we'll return the per-page average mapcount within the folio, except when the average is 0 but the folio is mapped: then we return 1.
For hugetlb folios and for large folios that are fully mapped into all address spaces, there is no change.
As an alternative, we could simply return 0 for non-hugetlb large folios, or disable this legacy interface with CONFIG_NO_PAGE_MAPCOUNT.
But the information exposed by this interface can still be valuable, and frequently we deal with fully-mapped large folios where the average corresponds to the actual page mapcount. So we'll leave it like this for now and document the new behavior.
Note: this interface is likely not very relevant for performance. If ever required, we could try doing a rather expensive rmap walk to collect precisely how often this folio page is mapped.
--- Documentation/admin-guide/mm/pagemap.rst | 7 +++++- fs/proc/internal.h | 31 ++++++++++++++++++++++++ fs/proc/page.c | 19 ++++++++++++--- 3 files changed, 53 insertions(+), 4 deletions(-)
Documentation/admin-guide/mm/pagemap.rst +6 -1
@@ -42,7 +42,12 @@ There are four components to pagemap: skip over unmapped regions. * ``/proc/kpagecount``. This file contains a 64-bit count of the number of - times each page is mapped, indexed by PFN. + times each page is mapped, indexed by PFN. Some kernel configurations do + not track the precise number of times a page part of a larger allocation + (e.g., THP) is mapped. In these configurations, the average number of + mappings per page in this larger allocation is returned instead. However, + if any page of the large allocation is mapped, the returned value will + be at least 1. The page-types tool in the tools/mm directory can be used to query the number of times a page is mapped.
fs/proc/internal.h +31 -0
@@ -174,6 +174,37 @@ static inline int folio_precise_page_mapcount(struct folio *folio,return mapcount; } +/** +* folio_average_page_mapcount() - Average number of mappings per page in this +* folio +* @folio: The folio. +* +* The average number of present user page table entries that reference each +* page in this folio as tracked via the RMAP: either referenced directly +* (PTE) or as part of a larger area that covers this page (e.g., PMD). +* +* Returns: The average number of mappings per page in this folio. 0 for +* folios that are not mapped to user space or are not tracked via the RMAP +* (e.g., shared zeropage). +*/ +static inline int folio_average_page_mapcount (struct folio *folio) +{ +int mapcount, entire_mapcount; +unsigned int adjust; + +if (!folio_test_large (folio)) +return atomic_read (&folio->_mapcount) +1 ; + + mapcount =folio_large_mapcount (folio); + entire_mapcount =folio_entire_mapcount (folio); +if (mapcount <= entire_mapcount) +return entire_mapcount; + mapcount -= entire_mapcount; + + adjust =folio_large_nr_pages (folio) /2 ; +return ((mapcount + adjust) >>folio_large_order (folio)) + + entire_mapcount; +}/* * array.c */
fs/proc/page.c +16 -3
@@ -67,9 +67,22 @@ static ssize_t kpagecount_read(struct file *file, char __user *buf,* memmaps that were actually initialized. */ page =pfn_to_online_page (pfn); -if (page) - mapcount =folio_precise_page_mapcount (page_folio (page), - page); +if (page) { +struct folio *folio =page_folio (page); + +if (IS_ENABLED (CONFIG_PAGE_MAPCOUNT )) { + mapcount =folio_precise_page_mapcount (folio, page); + }else { +/* +* Indicate the per-page average, but at least "1" for +* mapped folios. +*/ + mapcount =folio_average_page_mapcount (folio); +if (!mapcount &&folio_test_large (folio) && +folio_mapped (folio)) + mapcount =1 ; + } + }if (put_user (mapcount, out)) { ret = -EFAULT ;
Signature · 1 line
2.48.1
Zi Yan
reviewerRe: [PATCH v2 16/20] fs/proc/page: remove per-page mapcount dependency for /proc/kpagecount (CONFIG_NO_PAGE_MAPCOUNT)
Quoted 73 lines · David Hildenbrand
Let's implement an alternative when per-page mapcounts in large folios are no longer maintained -- soon with
CONFIG_NO_PAGE_MAPCOUNT.For large folios, we'll return the per-page average mapcount within the folio, except when the average is 0 but the folio is mapped: then we return 1.
For hugetlb folios and for large folios that are fully mapped into all address spaces, there is no change.
As an alternative, we could simply return 0 for non-hugetlb large folios, or disable this legacy interface with
CONFIG_NO_PAGE_MAPCOUNT.But the information exposed by this interface can still be valuable, and frequently we deal with fully-mapped large folios where the average corresponds to the actual page mapcount. So we'll leave it like this for now and document the new behavior.
Note: this interface is likely not very relevant for performance. If ever required, we could try doing a rather expensive rmap walk to collect precisely how often this folio page is mapped.
Signed-off-by David Hildenbrand <[email protected]>--- Documentation/admin-guide/mm/pagemap.rst | 7 +++++- fs/proc/internal.h | 31 ++++++++++++++++++++++++ fs/proc/page.c | 19 ++++++++++++--- 3 files changed, 53 insertions(+), 4 deletions(-)Documentation/admin-guide/mm/pagemap.rst +6
-1@@ -42,7 +42,12 @@ There are four components to pagemap: skip over unmapped regions. * ``/proc/kpagecount``. This file contains a 64-bit count of the number of - times each page is mapped, indexed by PFN. + times each page is mapped, indexed by PFN. Some kernel configurations do + not track the precise number of times a page part of a larger allocation + (e.g., THP) is mapped. In these configurations, the average number of + mappings per page in this larger allocation is returned instead. However, + if any page of the large allocation is mapped, the returned value will + be at least 1. The page-types tool in the tools/mm directory can be used to query the number of times a page is mapped.fs/proc/internal.h +18
-0@@ -174,6 +174,37 @@ static inline int folio_precise_page_mapcount(struct folio *folio,return mapcount; } +/** +* folio_average_page_mapcount() - Average number of mappings per page in this +* folio +* @folio: The folio. +* +* The average number of present user page table entries that reference each +* page in this folio as tracked via the RMAP: either referenced directly +* (PTE) or as part of a larger area that covers this page (e.g., PMD). +* +* Returns: The average number of mappings per page in this folio. 0 for +* folios that are not mapped to user space or are not tracked via the RMAP +* (e.g., shared zeropage). +*/ +static inline int folio_average_page_mapcount (struct folio *folio) +{ +int mapcount, entire_mapcount; +unsigned int adjust; +
fs/proc/internal.h static inline int folio_precise_page_mapcount(struct folio *folio, · line 204
Quoted 10 lines
fs/proc/internal.h +10
-0+if (!folio_test_large (folio)) +return atomic_read (&folio->_mapcount) +1 ; + + mapcount =folio_large_mapcount (folio); + entire_mapcount =folio_entire_mapcount (folio); +if (mapcount <= entire_mapcount) +return entire_mapcount; + mapcount -= entire_mapcount; + + adjust =folio_large_nr_pages (folio) /2 ;
Is there any reason for choosing this adjust number? A comment might be helpful in case people want to change it later, either with some reasoning or just saying it is chosen empirically.
Quoted 21 lines
fs/proc/internal.h +3
-0+return ((mapcount + adjust) >>folio_large_order (folio)) + + entire_mapcount; +}/* * array.c */ fs/proc/page.c +4
-3@@ -67,9 +67,22 @@ static ssize_t kpagecount_read(struct file *file, char __user *buf,* memmaps that were actually initialized. */ page =pfn_to_online_page (pfn); -if (page) - mapcount =folio_precise_page_mapcount (page_folio (page), - page); +if (page) { +struct folio *folio =page_folio (page); + +if (IS_ENABLED (CONFIG_PAGE_MAPCOUNT )) {
fs/proc/page.c static ssize_t kpagecount_read(struct file *file, char __user *buf, · line 83
Quoted 10 lines
fs/proc/page.c +10
-0+ mapcount =folio_precise_page_mapcount (folio, page); + }else { +/* +* Indicate the per-page average, but at least "1" for +* mapped folios. +*/ + mapcount =folio_average_page_mapcount (folio); +if (!mapcount &&folio_test_large (folio) && +folio_mapped (folio)) + mapcount =1 ;
This should be part of folio_average_page_mapcount() right? Otherwise, the comment on folio_average_page_mapcount() is not correct, since it can return 0 when a folio is mapped to user space.
Quoted 5 lines
fs/proc/page.c +2
-0+ } + }if (put_user (mapcount, out)) { ret = -EFAULT ;
Signature · 2 lines
Best Regards, Yan, Zi
Re: [PATCH v2 16/20] fs/proc/page: remove per-page mapcount dependency for /proc/kpagecount (CONFIG_NO_PAGE_MAPCOUNT)
Quoted 76 lines · Zi Yan
Quoted 75 lines · David Hildenbrand
Let's implement an alternative when per-page mapcounts in large folios are no longer maintained -- soon with
CONFIG_NO_PAGE_MAPCOUNT.For large folios, we'll return the per-page average mapcount within the folio, except when the average is 0 but the folio is mapped: then we return 1.
For hugetlb folios and for large folios that are fully mapped into all address spaces, there is no change.
As an alternative, we could simply return 0 for non-hugetlb large folios, or disable this legacy interface with
CONFIG_NO_PAGE_MAPCOUNT.But the information exposed by this interface can still be valuable, and frequently we deal with fully-mapped large folios where the average corresponds to the actual page mapcount. So we'll leave it like this for now and document the new behavior.
Note: this interface is likely not very relevant for performance. If ever required, we could try doing a rather expensive rmap walk to collect precisely how often this folio page is mapped.
Signed-off-by David Hildenbrand <[email protected]>--- Documentation/admin-guide/mm/pagemap.rst | 7 +++++- fs/proc/internal.h | 31 ++++++++++++++++++++++++ fs/proc/page.c | 19 ++++++++++++--- 3 files changed, 53 insertions(+), 4 deletions(-)Documentation/admin-guide/mm/pagemap.rst +6
-1@@ -42,7 +42,12 @@ There are four components to pagemap: skip over unmapped regions. * ``/proc/kpagecount``. This file contains a 64-bit count of the number of - times each page is mapped, indexed by PFN. + times each page is mapped, indexed by PFN. Some kernel configurations do + not track the precise number of times a page part of a larger allocation + (e.g., THP) is mapped. In these configurations, the average number of + mappings per page in this larger allocation is returned instead. However, + if any page of the large allocation is mapped, the returned value will + be at least 1. The page-types tool in the tools/mm directory can be used to query the number of times a page is mapped.fs/proc/internal.h +20
-0@@ -174,6 +174,37 @@ static inline int folio_precise_page_mapcount(struct folio *folio,return mapcount; } +/** +* folio_average_page_mapcount() - Average number of mappings per page in this +* folio +* @folio: The folio. +* +* The average number of present user page table entries that reference each +* page in this folio as tracked via the RMAP: either referenced directly +* (PTE) or as part of a larger area that covers this page (e.g., PMD). +* +* Returns: The average number of mappings per page in this folio. 0 for +* folios that are not mapped to user space or are not tracked via the RMAP +* (e.g., shared zeropage). +*/ +static inline int folio_average_page_mapcount (struct folio *folio) +{ +int mapcount, entire_mapcount; +unsigned int adjust; + +if (!folio_test_large (folio)) +return atomic_read (&folio->_mapcount) +1 ;
Quoted 8 lines
fs/proc/internal.h static inline int folio_precise_page_mapcount(struct folio *folio, · line 204
Quoted 8 lines
fs/proc/internal.h +8
-0+ + mapcount =folio_large_mapcount (folio); + entire_mapcount =folio_entire_mapcount (folio); +if (mapcount <= entire_mapcount) +return entire_mapcount; + mapcount -= entire_mapcount; + + adjust =folio_large_nr_pages (folio) /2 ;
Thanks for the review!
Quoted 4 lines
Is there any reason for choosing this adjust number? A comment might be helpful in case people want to change it later, either with some reasoning or just saying it is chosen empirically.
We're dividing by folio_large_nr_pages(folio) (shifting by folio_large_order(folio)), so this is not a magic number at all.
So this should be "ordinary" rounding.
Assume nr_pages = 512.
With 255 we want to round down, with 256 we want to round up.
255 / 512 = 0 :)
256 / 512 = 0 :(
Compared to:
(255 + (512 / 2)) / 512 = (255 + 256) / 512 = 0 :) (256 + (512 / 2)) / 512 = (256 + 256) / 512 = 1 :)
Quoted 26 lines
Quoted 26 lines
fs/proc/internal.h +3
-0+return ((mapcount + adjust) >>folio_large_order (folio)) + + entire_mapcount; +}/* * array.c */ fs/proc/page.c +8
-3@@ -67,9 +67,22 @@ static ssize_t kpagecount_read(struct file *file, char __user *buf,* memmaps that were actually initialized. */ page =pfn_to_online_page (pfn); -if (page) - mapcount =folio_precise_page_mapcount (page_folio (page), - page); +if (page) { +struct folio *folio =page_folio (page); + +if (IS_ENABLED (CONFIG_PAGE_MAPCOUNT )) { + mapcount =folio_precise_page_mapcount (folio, page); + }else { +/* +* Indicate the per-page average, but at least "1" for
Quoted 8 lines
fs/proc/page.c static ssize_t kpagecount_read(struct file *file, char __user *buf, · line 83
Quoted 6 lines
fs/proc/page.c +6
-0+* mapped folios. +*/ + mapcount =folio_average_page_mapcount (folio); +if (!mapcount &&folio_test_large (folio) && +folio_mapped (folio)) + mapcount =1 ;This should be part of
folio_average_page_mapcount()right?
No, that's not desired.
Quoted 2 lines
Otherwise, the comment on
folio_average_page_mapcount()is not correct, since it can return 0 when a folio is mapped to user space.
It's misleading. I'll clarify the comment, probably simply saying:
Returns: The average number of mappings per page in this folio.
Thanks!
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v2 16/20] fs/proc/page: remove per-page mapcount dependency for /proc/kpagecount (CONFIG_NO_PAGE_MAPCOUNT)
Quoted 102 lines · David Hildenbrand
Quoted 76 lines · Zi Yan
Quoted 75 lines · David Hildenbrand
Let's implement an alternative when per-page mapcounts in large folios are no longer maintained -- soon with
CONFIG_NO_PAGE_MAPCOUNT.For large folios, we'll return the per-page average mapcount within the folio, except when the average is 0 but the folio is mapped: then we return 1.
For hugetlb folios and for large folios that are fully mapped into all address spaces, there is no change.
As an alternative, we could simply return 0 for non-hugetlb large folios, or disable this legacy interface with
CONFIG_NO_PAGE_MAPCOUNT.But the information exposed by this interface can still be valuable, and frequently we deal with fully-mapped large folios where the average corresponds to the actual page mapcount. So we'll leave it like this for now and document the new behavior.
Note: this interface is likely not very relevant for performance. If ever required, we could try doing a rather expensive rmap walk to collect precisely how often this folio page is mapped.
Signed-off-by David Hildenbrand <[email protected]>--- Documentation/admin-guide/mm/pagemap.rst | 7 +++++- fs/proc/internal.h | 31 ++++++++++++++++++++++++ fs/proc/page.c | 19 ++++++++++++--- 3 files changed, 53 insertions(+), 4 deletions(-)Documentation/admin-guide/mm/pagemap.rst +6
-1@@ -42,7 +42,12 @@ There are four components to pagemap: skip over unmapped regions. * ``/proc/kpagecount``. This file contains a 64-bit count of the number of - times each page is mapped, indexed by PFN. + times each page is mapped, indexed by PFN. Some kernel configurations do + not track the precise number of times a page part of a larger allocation + (e.g., THP) is mapped. In these configurations, the average number of + mappings per page in this larger allocation is returned instead. However, + if any page of the large allocation is mapped, the returned value will + be at least 1. The page-types tool in the tools/mm directory can be used to query the number of times a page is mapped.fs/proc/internal.h +20
-0@@ -174,6 +174,37 @@ static inline int folio_precise_page_mapcount(struct folio *folio,return mapcount; } +/** +* folio_average_page_mapcount() - Average number of mappings per page in this +* folio +* @folio: The folio. +* +* The average number of present user page table entries that reference each +* page in this folio as tracked via the RMAP: either referenced directly +* (PTE) or as part of a larger area that covers this page (e.g., PMD). +* +* Returns: The average number of mappings per page in this folio. 0 for +* folios that are not mapped to user space or are not tracked via the RMAP +* (e.g., shared zeropage). +*/ +static inline int folio_average_page_mapcount (struct folio *folio) +{ +int mapcount, entire_mapcount; +unsigned int adjust; + +if (!folio_test_large (folio)) +return atomic_read (&folio->_mapcount) +1 ;Quoted 8 lines
fs/proc/internal.h static inline int folio_precise_page_mapcount(struct folio *folio, · line 204
Quoted 8 lines
fs/proc/internal.h +8
-0+ + mapcount =folio_large_mapcount (folio); + entire_mapcount =folio_entire_mapcount (folio); +if (mapcount <= entire_mapcount) +return entire_mapcount; + mapcount -= entire_mapcount; + + adjust =folio_large_nr_pages (folio) /2 ;Thanks for the review!
Quoted 4 lines
Is there any reason for choosing this adjust number? A comment might be helpful in case people want to change it later, either with some reasoning or just saying it is chosen empirically.
We're dividing by
folio_large_nr_pages(folio)(shifting byfolio_large_order(folio)), so this is not a magic number at all.So this should be "ordinary" rounding.
Assume
nr_pages= 512.With 255 we want to round down, with 256 we want to round up.
Quoted 7 lines
255 / 512 = 0 :)
256 / 512 = 0 :(Compared to:
(255 + (512 / 2)) / 512 = (255 + 256) / 512 = 0 :) (256 + (512 / 2)) / 512 = (256 + 256) / 512 = 1 :)
I think adding to the function doc:
"The average is calculated by rounding to the nearest integer."
might make it clearer.
Signature · 3 lines
Cheers, David / dhildenb
Zi Yan
reviewerRe: [PATCH v2 16/20] fs/proc/page: remove per-page mapcount dependency for /proc/kpagecount (CONFIG_NO_PAGE_MAPCOUNT)
In reply to David HildenbrandQuoted 86 lines · David Hildenbrand
Quoted 73 lines · Zi Yan
Quoted 72 lines · David Hildenbrand
Let's implement an alternative when per-page mapcounts in large folios are no longer maintained -- soon with
CONFIG_NO_PAGE_MAPCOUNT.For large folios, we'll return the per-page average mapcount within the folio, except when the average is 0 but the folio is mapped: then we return 1.
For hugetlb folios and for large folios that are fully mapped into all address spaces, there is no change.
As an alternative, we could simply return 0 for non-hugetlb large folios, or disable this legacy interface with
CONFIG_NO_PAGE_MAPCOUNT.But the information exposed by this interface can still be valuable, and frequently we deal with fully-mapped large folios where the average corresponds to the actual page mapcount. So we'll leave it like this for now and document the new behavior.
Note: this interface is likely not very relevant for performance. If ever required, we could try doing a rather expensive rmap walk to collect precisely how often this folio page is mapped.
Signed-off-by David Hildenbrand <[email protected]>--- Documentation/admin-guide/mm/pagemap.rst | 7 +++++- fs/proc/internal.h | 31 ++++++++++++++++++++++++ fs/proc/page.c | 19 ++++++++++++--- 3 files changed, 53 insertions(+), 4 deletions(-)Documentation/admin-guide/mm/pagemap.rst +6
-1@@ -42,7 +42,12 @@ There are four components to pagemap: skip over unmapped regions. * ``/proc/kpagecount``. This file contains a 64-bit count of the number of - times each page is mapped, indexed by PFN. + times each page is mapped, indexed by PFN. Some kernel configurations do + not track the precise number of times a page part of a larger allocation + (e.g., THP) is mapped. In these configurations, the average number of + mappings per page in this larger allocation is returned instead. However, + if any page of the large allocation is mapped, the returned value will + be at least 1. The page-types tool in the tools/mm directory can be used to query the number of times a page is mapped.fs/proc/internal.h +19
-0@@ -174,6 +174,37 @@ static inline int folio_precise_page_mapcount(struct folio *folio,return mapcount; } +/** +* folio_average_page_mapcount() - Average number of mappings per page in this +* folio +* @folio: The folio. +* +* The average number of present user page table entries that reference each +* page in this folio as tracked via the RMAP: either referenced directly +* (PTE) or as part of a larger area that covers this page (e.g., PMD). +* +* Returns: The average number of mappings per page in this folio. 0 for +* folios that are not mapped to user space or are not tracked via the RMAP +* (e.g., shared zeropage). +*/ +static inline int folio_average_page_mapcount(struct folio *folio) +{ +int mapcount, entire_mapcount; +unsigned int adjust; + +if (!folio_test_large (folio)) +return atomic_read (&folio->_mapcount) +1 ;Quoted 8 lines
fs/proc/internal.h static inline int folio_precise_page_mapcount(struct folio *folio, · line 204
Quoted 8 lines
fs/proc/internal.h +8
-0+ + mapcount =folio_large_mapcount (folio); + entire_mapcount =folio_entire_mapcount (folio); +if (mapcount <= entire_mapcount) +return entire_mapcount; + mapcount -= entire_mapcount; + + adjust =folio_large_nr_pages (folio) /2 ;Thanks for the review!
Quoted 1 line
Quoted 7 lines
Quoted 3 lines
Is there any reason for choosing this adjust number? A comment might be helpful in case people want to change it later, either with some reasoning or just saying it is chosen empirically.
We're dividing by
folio_large_nr_pages(folio)(shifting byfolio_large_order(folio)), so this is not a magic number at all.So this should be "ordinary" rounding.
I thought the rounding would be (mapcount + 511) / 512. But that means if one subpage is mapped, the average will be 1. Your rounding means if at least half of the subpages is mapped, the average will be 1. Others might think 1/3 is mapped, the average will be 1. That is why I think adjust looks like a magic number.
Quoted 48 lines
Assume
nr_pages= 512.With 255 we want to round down, with 256 we want to round up.
255 / 512 = 0 :)
256 / 512 = 0 :(Compared to:
(255 + (512 / 2)) / 512 = (255 + 256) / 512 = 0 :) (256 + (512 / 2)) / 512 = (256 + 256) / 512 = 1 :)
Quoted 26 lines
Quoted 26 lines
fs/proc/internal.h +3
-0+return ((mapcount + adjust) >>folio_large_order (folio)) + + entire_mapcount; +}/* * array.c */ fs/proc/page.c +8
-3@@ -67,9 +67,22 @@ static ssize_t kpagecount_read(struct file *file, char __user *buf,* memmaps that were actually initialized. */ page =pfn_to_online_page (pfn); -if (page) - mapcount =folio_precise_page_mapcount (page_folio (page), - page); +if (page) { +struct folio *folio =page_folio (page); + +if (IS_ENABLED (CONFIG_PAGE_MAPCOUNT )) { + mapcount =folio_precise_page_mapcount (folio, page); + }else { +/* +* Indicate the per-page average, but at least "1" for Quoted 8 lines
fs/proc/page.c static ssize_t kpagecount_read(struct file *file, char __user *buf, · line 83
Quoted 6 lines
fs/proc/page.c +6
-0+* mapped folios. +*/ + mapcount =folio_average_page_mapcount (folio); +if (!mapcount &&folio_test_large (folio) && +folio_mapped (folio)) + mapcount =1 ;This should be part of
folio_average_page_mapcount()right?
Quoted 8 lines
No, that's not desired.
Quoted 2 lines
Otherwise, the comment on
folio_average_page_mapcount()is not correct, since it can return 0 when a folio is mapped to user space.It's misleading. I'll clarify the comment, probably simply saying:
Returns: The average number of mappings per page in this folio.
Got it.
Best Regards,
Yan, Zi
Re: [PATCH v2 16/20] fs/proc/page: remove per-page mapcount dependency for /proc/kpagecount (CONFIG_NO_PAGE_MAPCOUNT)
Quoted 92 lines · Zi Yan
Quoted 90 lines · David Hildenbrand
Quoted 73 lines · Zi Yan
Quoted 72 lines · David Hildenbrand
Let's implement an alternative when per-page mapcounts in large folios are no longer maintained -- soon with
CONFIG_NO_PAGE_MAPCOUNT.For large folios, we'll return the per-page average mapcount within the folio, except when the average is 0 but the folio is mapped: then we return 1.
For hugetlb folios and for large folios that are fully mapped into all address spaces, there is no change.
As an alternative, we could simply return 0 for non-hugetlb large folios, or disable this legacy interface with
CONFIG_NO_PAGE_MAPCOUNT.But the information exposed by this interface can still be valuable, and frequently we deal with fully-mapped large folios where the average corresponds to the actual page mapcount. So we'll leave it like this for now and document the new behavior.
Note: this interface is likely not very relevant for performance. If ever required, we could try doing a rather expensive rmap walk to collect precisely how often this folio page is mapped.
Signed-off-by David Hildenbrand <[email protected]>--- Documentation/admin-guide/mm/pagemap.rst | 7 +++++- fs/proc/internal.h | 31 ++++++++++++++++++++++++ fs/proc/page.c | 19 ++++++++++++--- 3 files changed, 53 insertions(+), 4 deletions(-)Documentation/admin-guide/mm/pagemap.rst +6
-1@@ -42,7 +42,12 @@ There are four components to pagemap: skip over unmapped regions. * ``/proc/kpagecount``. This file contains a 64-bit count of the number of - times each page is mapped, indexed by PFN. + times each page is mapped, indexed by PFN. Some kernel configurations do + not track the precise number of times a page part of a larger allocation + (e.g., THP) is mapped. In these configurations, the average number of + mappings per page in this larger allocation is returned instead. However, + if any page of the large allocation is mapped, the returned value will + be at least 1. The page-types tool in the tools/mm directory can be used to query the number of times a page is mapped.fs/proc/internal.h +19
-0@@ -174,6 +174,37 @@ static inline int folio_precise_page_mapcount(struct folio *folio,return mapcount; } +/** +* folio_average_page_mapcount() - Average number of mappings per page in this +* folio +* @folio: The folio. +* +* The average number of present user page table entries that reference each +* page in this folio as tracked via the RMAP: either referenced directly +* (PTE) or as part of a larger area that covers this page (e.g., PMD). +* +* Returns: The average number of mappings per page in this folio. 0 for +* folios that are not mapped to user space or are not tracked via the RMAP +* (e.g., shared zeropage). +*/ +static inline int folio_average_page_mapcount(struct folio *folio) +{ +int mapcount, entire_mapcount; +unsigned int adjust; + +if (!folio_test_large (folio)) +return atomic_read (&folio->_mapcount) +1 ;Quoted 8 lines
fs/proc/internal.h static inline int folio_precise_page_mapcount(struct folio *folio, · line 204
Quoted 8 lines
fs/proc/internal.h +8
-0+ + mapcount =folio_large_mapcount (folio); + entire_mapcount =folio_entire_mapcount (folio); +if (mapcount <= entire_mapcount) +return entire_mapcount; + mapcount -= entire_mapcount; + + adjust =folio_large_nr_pages (folio) /2 ;Thanks for the review!
Quoted 4 lines
Is there any reason for choosing this adjust number? A comment might be helpful in case people want to change it later, either with some reasoning or just saying it is chosen empirically.
Quoted 5 lines
Quoted 3 lines
We're dividing by
folio_large_nr_pages(folio)(shifting byfolio_large_order(folio)), so this is not a magic number at all.So this should be "ordinary" rounding.
I thought the rounding would be (mapcount + 511) / 512.
Yes, that's "rounding up".
Quoted 6 lines
But
that means if one subpage is mapped, the average will be 1. Your rounding means if at least half of the subpages is mapped, the average will be 1. Others might think 1/3 is mapped, the average will be 1. That is why I think adjust looks like a magic number.
I think all callers could tolerate (or benefit) from folio_average_page_mapcount() returning at least 1 in case any page is mapped.
There was a reason why I decided to round to the nearest integer instead.
Let me think about this once more, I went back and forth a couple of times on this.
Signature · 3 lines
Cheers, David / dhildenb
Zi Yan
reviewerRe: [PATCH v2 16/20] fs/proc/page: remove per-page mapcount dependency for /proc/kpagecount (CONFIG_NO_PAGE_MAPCOUNT)
Quoted 108 lines · David Hildenbrand
Quoted 92 lines · Zi Yan
Quoted 90 lines · David Hildenbrand
Quoted 73 lines · Zi Yan
Quoted 72 lines · David Hildenbrand
Let's implement an alternative when per-page mapcounts in large folios are no longer maintained -- soon with
CONFIG_NO_PAGE_MAPCOUNT.For large folios, we'll return the per-page average mapcount within the folio, except when the average is 0 but the folio is mapped: then we return 1.
For hugetlb folios and for large folios that are fully mapped into all address spaces, there is no change.
As an alternative, we could simply return 0 for non-hugetlb large folios, or disable this legacy interface with
CONFIG_NO_PAGE_MAPCOUNT.But the information exposed by this interface can still be valuable, and frequently we deal with fully-mapped large folios where the average corresponds to the actual page mapcount. So we'll leave it like this for now and document the new behavior.
Note: this interface is likely not very relevant for performance. If ever required, we could try doing a rather expensive rmap walk to collect precisely how often this folio page is mapped.
Signed-off-by David Hildenbrand <[email protected]>--- Documentation/admin-guide/mm/pagemap.rst | 7 +++++- fs/proc/internal.h | 31 ++++++++++++++++++++++++ fs/proc/page.c | 19 ++++++++++++--- 3 files changed, 53 insertions(+), 4 deletions(-)Documentation/admin-guide/mm/pagemap.rst +6
-1@@ -42,7 +42,12 @@ There are four components to pagemap: skip over unmapped regions. * ``/proc/kpagecount``. This file contains a 64-bit count of the number of - times each page is mapped, indexed by PFN. + times each page is mapped, indexed by PFN. Some kernel configurations do + not track the precise number of times a page part of a larger allocation + (e.g., THP) is mapped. In these configurations, the average number of + mappings per page in this larger allocation is returned instead. However, + if any page of the large allocation is mapped, the returned value will + be at least 1. The page-types tool in the tools/mm directory can be used to query the number of times a page is mapped.fs/proc/internal.h +19
-0@@ -174,6 +174,37 @@ static inline int folio_precise_page_mapcount(struct folio *folio,return mapcount; } +/** +* folio_average_page_mapcount() - Average number of mappings per page in this +* folio +* @folio: The folio. +* +* The average number of present user page table entries that reference each +* page in this folio as tracked via the RMAP: either referenced directly +* (PTE) or as part of a larger area that covers this page (e.g., PMD). +* +* Returns: The average number of mappings per page in this folio. 0 for +* folios that are not mapped to user space or are not tracked via the RMAP +* (e.g., shared zeropage). +*/ +static inline int folio_average_page_mapcount(struct folio *folio) +{ +int mapcount, entire_mapcount; +unsigned int adjust; + +if (!folio_test_large (folio)) +return atomic_read (&folio->_mapcount) +1 ;Quoted 8 lines
fs/proc/internal.h static inline int folio_precise_page_mapcount(struct folio *folio, · line 204
Quoted 8 lines
fs/proc/internal.h +8
-0+ + mapcount =folio_large_mapcount (folio); + entire_mapcount =folio_entire_mapcount (folio); +if (mapcount <= entire_mapcount) +return entire_mapcount; + mapcount -= entire_mapcount; + + adjust =folio_large_nr_pages (folio) /2 ;Thanks for the review!
Quoted 4 lines
Is there any reason for choosing this adjust number? A comment might be helpful in case people want to change it later, either with some reasoning or just saying it is chosen empirically.
Quoted 5 lines
Quoted 3 lines
We're dividing by
folio_large_nr_pages(folio)(shifting byfolio_large_order(folio)), so this is not a magic number at all.So this should be "ordinary" rounding.
I thought the rounding would be (mapcount + 511) / 512.
Yes, that's "rounding up".
Quoted 6 lines
But
that means if one subpage is mapped, the average will be 1. Your rounding means if at least half of the subpages is mapped, the average will be 1. Others might think 1/3 is mapped, the average will be 1. That is why I think adjust looks like a magic number.
Quoted 5 lines
I think all callers could tolerate (or benefit) from
folio_average_page_mapcount()returning at least 1 in case any page is mapped.There was a reason why I decided to round to the nearest integer instead.
Let me think about this once more, I went back and forth a couple of times on this.
Sure. Your current choice might be good enough for now. My intend of adding a comment here is just to let people know the adjust can be changed in the future. :)
Best Regards,
Yan, Zi
Re: [PATCH v2 16/20] fs/proc/page: remove per-page mapcount dependency for /proc/kpagecount (CONFIG_NO_PAGE_MAPCOUNT)
Quoted 112 lines · Zi Yan
Quoted 110 lines · David Hildenbrand
Quoted 92 lines · Zi Yan
Quoted 90 lines · David Hildenbrand
Quoted 73 lines · Zi Yan
Quoted 72 lines · David Hildenbrand
Let's implement an alternative when per-page mapcounts in large folios are no longer maintained -- soon with
CONFIG_NO_PAGE_MAPCOUNT.For large folios, we'll return the per-page average mapcount within the folio, except when the average is 0 but the folio is mapped: then we return 1.
For hugetlb folios and for large folios that are fully mapped into all address spaces, there is no change.
As an alternative, we could simply return 0 for non-hugetlb large folios, or disable this legacy interface with
CONFIG_NO_PAGE_MAPCOUNT.But the information exposed by this interface can still be valuable, and frequently we deal with fully-mapped large folios where the average corresponds to the actual page mapcount. So we'll leave it like this for now and document the new behavior.
Note: this interface is likely not very relevant for performance. If ever required, we could try doing a rather expensive rmap walk to collect precisely how often this folio page is mapped.
Signed-off-by David Hildenbrand <[email protected]>--- Documentation/admin-guide/mm/pagemap.rst | 7 +++++- fs/proc/internal.h | 31 ++++++++++++++++++++++++ fs/proc/page.c | 19 ++++++++++++--- 3 files changed, 53 insertions(+), 4 deletions(-)Documentation/admin-guide/mm/pagemap.rst +6
-1@@ -42,7 +42,12 @@ There are four components to pagemap: skip over unmapped regions. * ``/proc/kpagecount``. This file contains a 64-bit count of the number of - times each page is mapped, indexed by PFN. + times each page is mapped, indexed by PFN. Some kernel configurations do + not track the precise number of times a page part of a larger allocation + (e.g., THP) is mapped. In these configurations, the average number of + mappings per page in this larger allocation is returned instead. However, + if any page of the large allocation is mapped, the returned value will + be at least 1. The page-types tool in the tools/mm directory can be used to query the number of times a page is mapped.fs/proc/internal.h +19
-0@@ -174,6 +174,37 @@ static inline int folio_precise_page_mapcount(struct folio *folio,return mapcount; } +/** +* folio_average_page_mapcount() - Average number of mappings per page in this +* folio +* @folio: The folio. +* +* The average number of present user page table entries that reference each +* page in this folio as tracked via the RMAP: either referenced directly +* (PTE) or as part of a larger area that covers this page (e.g., PMD). +* +* Returns: The average number of mappings per page in this folio. 0 for +* folios that are not mapped to user space or are not tracked via the RMAP +* (e.g., shared zeropage). +*/ +static inline int folio_average_page_mapcount(struct folio *folio) +{ +int mapcount, entire_mapcount; +unsigned int adjust; + +if (!folio_test_large (folio)) +return atomic_read (&folio->_mapcount) +1 ;Quoted 8 lines
fs/proc/internal.h static inline int folio_precise_page_mapcount(struct folio *folio, · line 204
Quoted 8 lines
fs/proc/internal.h +8
-0+ + mapcount =folio_large_mapcount (folio); + entire_mapcount =folio_entire_mapcount (folio); +if (mapcount <= entire_mapcount) +return entire_mapcount; + mapcount -= entire_mapcount; + + adjust =folio_large_nr_pages (folio) /2 ;Thanks for the review!
Quoted 4 lines
Is there any reason for choosing this adjust number? A comment might be helpful in case people want to change it later, either with some reasoning or just saying it is chosen empirically.
Quoted 5 lines
Quoted 3 lines
We're dividing by
folio_large_nr_pages(folio)(shifting byfolio_large_order(folio)), so this is not a magic number at all.So this should be "ordinary" rounding.
I thought the rounding would be (mapcount + 511) / 512.
Yes, that's "rounding up".
Quoted 6 lines
But
that means if one subpage is mapped, the average will be 1. Your rounding means if at least half of the subpages is mapped, the average will be 1. Others might think 1/3 is mapped, the average will be 1. That is why I think adjust looks like a magic number.I think all callers could tolerate (or benefit) from
folio_average_page_mapcount()returning at least 1 in case any page is mapped.
Quoted 7 lines
Quoted 3 lines
There was a reason why I decided to round to the nearest integer instead.
Let me think about this once more, I went back and forth a couple of times on this.
Sure. Your current choice might be good enough for now. My intend of adding a comment here is just to let people know the adjust can be changed in the future. :)
The following will make the callers easier to read, while keeping the rounding to the next integer for the other cases untouched.
+/**
+ * folio_average_page_mapcount() - Average number of mappings per page in this
+ * folio
+ * @folio: The folio.
+ *
+ * The average number of present user page table entries that reference each
+ * page in this folio as tracked via the RMAP: either referenced directly
+ * (PTE) or as part of a larger area that covers this page (e.g., PMD).
+ *
+ * The average is calculated by rounding to the nearest integer; however,
+ * if at least a single page is mapped, the average will be at least 1.
+ *
+ * Returns: The average number of mappings per page in this folio.
+ */
+static inline int folio_average_page_mapcount(struct folio *folio) +{
+ int mapcount, entire_mapcount, avg;
+
+ if (!folio_test_large(folio))
+ return atomic_read(&folio->_mapcount) + 1;
+
+ mapcount = folio_large_mapcount(folio);
+ if (unlikely(mapcount <= 0))
+ return 0;
+ entire_mapcount = folio_entire_mapcount(folio);
+ if (mapcount <= entire_mapcount)
+ return entire_mapcount;
+ mapcount -= entire_mapcount;
+
+ /* Round to closest integer ... */ + avg = (mapcount + folio_large_nr_pages(folio) / 2) >> folio_large_order(folio); + avg += entire_mapcount; + /* ... but return at least 1. */ + return max_t(int, avg, 1);
+}
Signature · 3 lines
Cheers, David / dhildenb
Zi Yan
reviewerRe: [PATCH v2 16/20] fs/proc/page: remove per-page mapcount dependency for /proc/kpagecount (CONFIG_NO_PAGE_MAPCOUNT)
Quoted 149 lines · David Hildenbrand
Quoted 112 lines · Zi Yan
Quoted 110 lines · David Hildenbrand
Quoted 92 lines · Zi Yan
Quoted 90 lines · David Hildenbrand
Quoted 73 lines · Zi Yan
Quoted 72 lines · David Hildenbrand
Let's implement an alternative when per-page mapcounts in large folios are no longer maintained -- soon with
CONFIG_NO_PAGE_MAPCOUNT.For large folios, we'll return the per-page average mapcount within the folio, except when the average is 0 but the folio is mapped: then we return 1.
For hugetlb folios and for large folios that are fully mapped into all address spaces, there is no change.
As an alternative, we could simply return 0 for non-hugetlb large folios, or disable this legacy interface with
CONFIG_NO_PAGE_MAPCOUNT.But the information exposed by this interface can still be valuable, and frequently we deal with fully-mapped large folios where the average corresponds to the actual page mapcount. So we'll leave it like this for now and document the new behavior.
Note: this interface is likely not very relevant for performance. If ever required, we could try doing a rather expensive rmap walk to collect precisely how often this folio page is mapped.
Signed-off-by David Hildenbrand <[email protected]>--- Documentation/admin-guide/mm/pagemap.rst | 7 +++++- fs/proc/internal.h | 31 ++++++++++++++++++++++++ fs/proc/page.c | 19 ++++++++++++--- 3 files changed, 53 insertions(+), 4 deletions(-)Documentation/admin-guide/mm/pagemap.rst +6
-1@@ -42,7 +42,12 @@ There are four components to pagemap: skip over unmapped regions. * ``/proc/kpagecount``. This file contains a 64-bit count of the number of - times each page is mapped, indexed by PFN. + times each page is mapped, indexed by PFN. Some kernel configurations do + not track the precise number of times a page part of a larger allocation + (e.g., THP) is mapped. In these configurations, the average number of + mappings per page in this larger allocation is returned instead. However, + if any page of the large allocation is mapped, the returned value will + be at least 1. The page-types tool in the tools/mm directory can be used to query the number of times a page is mapped.fs/proc/internal.h +19
-0@@ -174,6 +174,37 @@ static inline int folio_precise_page_mapcount(struct folio *folio,return mapcount; } +/** +* folio_average_page_mapcount() - Average number of mappings per page in this +* folio +* @folio: The folio. +* +* The average number of present user page table entries that reference each +* page in this folio as tracked via the RMAP: either referenced directly +* (PTE) or as part of a larger area that covers this page (e.g., PMD). +* +* Returns: The average number of mappings per page in this folio. 0 for +* folios that are not mapped to user space or are not tracked via the RMAP +* (e.g., shared zeropage). +*/ +static inline int folio_average_page_mapcount(struct folio *folio) +{ +int mapcount, entire_mapcount; +unsigned int adjust; + +if (!folio_test_large (folio)) +return atomic_read (&folio->_mapcount) +1 ;Quoted 8 lines
fs/proc/internal.h static inline int folio_precise_page_mapcount(struct folio *folio, · line 203
Quoted 8 lines
fs/proc/internal.h +8
-0+ + mapcount =folio_large_mapcount (folio); + entire_mapcount =folio_entire_mapcount (folio); +if (mapcount <= entire_mapcount) +return entire_mapcount; + mapcount -= entire_mapcount; + + adjust =folio_large_nr_pages (folio) /2 ;Thanks for the review!
Quoted 4 lines
Is there any reason for choosing this adjust number? A comment might be helpful in case people want to change it later, either with some reasoning or just saying it is chosen empirically.
Quoted 5 lines
Quoted 3 lines
We're dividing by
folio_large_nr_pages(folio)(shifting byfolio_large_order(folio)), so this is not a magic number at all.So this should be "ordinary" rounding.
I thought the rounding would be (mapcount + 511) / 512.
Yes, that's "rounding up".
Quoted 6 lines
But
that means if one subpage is mapped, the average will be 1. Your rounding means if at least half of the subpages is mapped, the average will be 1. Others might think 1/3 is mapped, the average will be 1. That is why I think adjust looks like a magic number.I think all callers could tolerate (or benefit) from
folio_average_page_mapcount()returning at least 1 in case any page is mapped.Quoted 7 lines
Quoted 3 lines
There was a reason why I decided to round to the nearest integer instead.
Let me think about this once more, I went back and forth a couple of times on this.
Sure. Your current choice might be good enough for now. My intend of adding a comment here is just to let people know the adjust can be changed in the future. :)
The following will make the callers easier to read, while keeping the rounding to the next integer for the other cases untouched.
(quoted diff) +25
-0+/** + * folio_average_page_mapcount() - Average number of mappings per page in this + * folio + * @folio: The folio. + * + * The average number of present user page table entries that reference each + * page in this folio as tracked via the RMAP: either referenced directly + * (PTE) or as part of a larger area that covers this page (e.g., PMD). + * + * The average is calculated by rounding to the nearest integer; however, + * if at least a single page is mapped, the average will be at least 1. + * + * Returns: The average number of mappings per page in this folio. + */ +static inline int folio_average_page_mapcount(struct folio *folio) +{ + int mapcount, entire_mapcount, avg; + + if (!folio_test_large(folio)) + return atomic_read(&folio->_mapcount) + 1; + + mapcount = folio_large_mapcount(folio); + if (unlikely(mapcount <= 0)) + return 0; + entire_mapcount = folio_entire_mapcount(folio);
Review
Quoted 10 lines
(quoted diff) +10
-0+ if (mapcount <= entire_mapcount) + return entire_mapcount; + mapcount -= entire_mapcount; + + /* Round to closest integer ... */ + avg = (mapcount + folio_large_nr_pages(folio) / 2) >> folio_large_order(folio); + avg += entire_mapcount; + /* ... but return at least 1. */ + return max_t(int, avg, 1); +}
LGTM. Thanks.
Best Regards,
Yan, Zi
Re: [PATCH v2 16/20] fs/proc/page: remove per-page mapcount dependency for /proc/kpagecount (CONFIG_NO_PAGE_MAPCOUNT)
Quoted 155 lines · Zi Yan
Quoted 153 lines · David Hildenbrand
Quoted 112 lines · Zi Yan
Quoted 110 lines · David Hildenbrand
Quoted 92 lines · Zi Yan
Quoted 90 lines · David Hildenbrand
Quoted 73 lines · Zi Yan
Quoted 72 lines · David Hildenbrand
Let's implement an alternative when per-page mapcounts in large folios are no longer maintained -- soon with
CONFIG_NO_PAGE_MAPCOUNT.For large folios, we'll return the per-page average mapcount within the folio, except when the average is 0 but the folio is mapped: then we return 1.
For hugetlb folios and for large folios that are fully mapped into all address spaces, there is no change.
As an alternative, we could simply return 0 for non-hugetlb large folios, or disable this legacy interface with
CONFIG_NO_PAGE_MAPCOUNT.But the information exposed by this interface can still be valuable, and frequently we deal with fully-mapped large folios where the average corresponds to the actual page mapcount. So we'll leave it like this for now and document the new behavior.
Note: this interface is likely not very relevant for performance. If ever required, we could try doing a rather expensive rmap walk to collect precisely how often this folio page is mapped.
Signed-off-by David Hildenbrand <[email protected]>--- Documentation/admin-guide/mm/pagemap.rst | 7 +++++- fs/proc/internal.h | 31 ++++++++++++++++++++++++ fs/proc/page.c | 19 ++++++++++++--- 3 files changed, 53 insertions(+), 4 deletions(-)Documentation/admin-guide/mm/pagemap.rst +6
-1@@ -42,7 +42,12 @@ There are four components to pagemap: skip over unmapped regions. * ``/proc/kpagecount``. This file contains a 64-bit count of the number of - times each page is mapped, indexed by PFN. + times each page is mapped, indexed by PFN. Some kernel configurations do + not track the precise number of times a page part of a larger allocation + (e.g., THP) is mapped. In these configurations, the average number of + mappings per page in this larger allocation is returned instead. However, + if any page of the large allocation is mapped, the returned value will + be at least 1. The page-types tool in the tools/mm directory can be used to query the number of times a page is mapped.fs/proc/internal.h +19
-0@@ -174,6 +174,37 @@ static inline int folio_precise_page_mapcount(struct folio *folio,return mapcount; } +/** +* folio_average_page_mapcount() - Average number of mappings per page in this +* folio +* @folio: The folio. +* +* The average number of present user page table entries that reference each +* page in this folio as tracked via the RMAP: either referenced directly +* (PTE) or as part of a larger area that covers this page (e.g., PMD). +* +* Returns: The average number of mappings per page in this folio. 0 for +* folios that are not mapped to user space or are not tracked via the RMAP +* (e.g., shared zeropage). +*/ +static inline int folio_average_page_mapcount(struct folio *folio) +{ +int mapcount, entire_mapcount; +unsigned int adjust; + +if (!folio_test_large (folio)) +return atomic_read (&folio->_mapcount) +1 ;Quoted 8 lines
fs/proc/internal.h static inline int folio_precise_page_mapcount(struct folio *folio, · line 203
Quoted 8 lines
fs/proc/internal.h +8
-0+ + mapcount =folio_large_mapcount (folio); + entire_mapcount =folio_entire_mapcount (folio); +if (mapcount <= entire_mapcount) +return entire_mapcount; + mapcount -= entire_mapcount; + + adjust =folio_large_nr_pages (folio) /2 ;Thanks for the review!
Quoted 4 lines
Is there any reason for choosing this adjust number? A comment might be helpful in case people want to change it later, either with some reasoning or just saying it is chosen empirically.
Quoted 5 lines
Quoted 3 lines
We're dividing by
folio_large_nr_pages(folio)(shifting byfolio_large_order(folio)), so this is not a magic number at all.So this should be "ordinary" rounding.
I thought the rounding would be (mapcount + 511) / 512.
Yes, that's "rounding up".
Quoted 6 lines
But
that means if one subpage is mapped, the average will be 1. Your rounding means if at least half of the subpages is mapped, the average will be 1. Others might think 1/3 is mapped, the average will be 1. That is why I think adjust looks like a magic number.I think all callers could tolerate (or benefit) from
folio_average_page_mapcount()returning at least 1 in case any page is mapped.Quoted 7 lines
Quoted 3 lines
There was a reason why I decided to round to the nearest integer instead.
Let me think about this once more, I went back and forth a couple of times on this.
Sure. Your current choice might be good enough for now. My intend of adding a comment here is just to let people know the adjust can be changed in the future. :)
The following will make the callers easier to read, while keeping the rounding to the next integer for the other cases untouched.
(quoted diff) +29
-0+/** + * folio_average_page_mapcount() - Average number of mappings per page in this + * folio + * @folio: The folio. + * + * The average number of present user page table entries that reference each + * page in this folio as tracked via the RMAP: either referenced directly + * (PTE) or as part of a larger area that covers this page (e.g., PMD). + * + * The average is calculated by rounding to the nearest integer; however, + * if at least a single page is mapped, the average will be at least 1. + * + * Returns: The average number of mappings per page in this folio. + */ +static inline int folio_average_page_mapcount(struct folio *folio) +{ + int mapcount, entire_mapcount, avg; + + if (!folio_test_large(folio)) + return atomic_read(&folio->_mapcount) + 1; + + mapcount = folio_large_mapcount(folio); + if (unlikely(mapcount <= 0)) + return 0; + entire_mapcount = folio_entire_mapcount(folio); + if (mapcount <= entire_mapcount) + return entire_mapcount; + mapcount -= entire_mapcount; +
Quoted 8 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+ /* Round to closest integer ... */ + avg = (mapcount + folio_large_nr_pages(folio) / 2) >> folio_large_order(folio); + avg += entire_mapcount; + /* ... but return at least 1. */ + return max_t(int, avg, 1); +}LGTM. Thanks.
Thanks! BTW, I think I chose the "round to closest integer" primarily to make the PSS estimate a bit better. But that is indeed something that can be adjusted easily later.
BTW, as commented in the cover letter, being able to calculate the avg without the entire_mapcount could clean this function up quite a bit (and make it completely atomic), but that will require more work.
Signature · 3 lines
Cheers, David / dhildenb
PATCH v2 17/20 fs/proc/task_mmu: remove per-page mapcount dependency for PM_MMAP_EXCLUSIVE (CONFIG_NO_PAGE_MAPCOUNT)
In reply to David HildenbrandLet's implement an alternative when per-page mapcounts in large folios are no longer maintained -- soon with CONFIG_NO_PAGE_MAPCOUNT.
PM_MMAP_EXCLUSIVE will now be set if folio_likely_mapped_shared() is true -- when the folio is considered "mapped shared", including when it once was "mapped shared" but no longer is, as documented.
This might result in and under-indication of "exclusively mapped", which is considered better than over-indicating it: under-estimating the USS (Unique Set Size) is better than over-estimating it.
As an alternative, we could simply remove that flag with CONFIG_NO_PAGE_MAPCOUNT completely, but there might be value to it. So, let's keep it like that and document the behavior.
--- Documentation/admin-guide/mm/pagemap.rst | 9 +++++++++ fs/proc/task_mmu.c | 11 +++++++++-- 2 files changed, 18 insertions(+), 2 deletions(-)
Documentation/admin-guide/mm/pagemap.rst +9 -0
@@ -37,6 +37,15 @@ There are four components to pagemap: precisely which pages are mapped (or in swap) and comparing mapped pages between processes. + Note that in some kernel configurations, all pages part of a larger + allocation (e.g., THP) might be considered "mapped shared" if the large + allocation is considered "mapped shared": if not all pages are exclusive to + the same process. Further, some kernel configurations might consider larger + allocations "mapped shared", if they were at one point considered + "mapped shared", even if they would now be considered "exclusively mapped". + Consequently, in these kernel configurations, bit 56 might be set although + the page is actually "exclusively mapped" + Efficient users of this interface will use ``/proc/pid/maps`` to determine which areas of memory are actually mapped and llseek to skip over unmapped regions.
fs/proc/task_mmu.c +9 -2
@@ -1651,6 +1651,13 @@ static int add_to_pagemap(pagemap_entry_t *pme, struct pagemapread *pm)return 0 ; } +static bool __folio_page_mapped_exclusively (struct folio *folio,struct page *page) +{ +if (IS_ENABLED (CONFIG_PAGE_MAPCOUNT )) +return folio_precise_page_mapcount (folio, page) ==1 ; +return !folio_maybe_mapped_shared (folio); +} +static int pagemap_pte_hole (unsigned long start,unsigned long end,__always_unused int depth,struct mm_walk *walk) {
@@ -1739,7 +1746,7 @@ static pagemap_entry_t pte_to_pagemap_entry(struct pagemapread *pm,if (!folio_test_anon (folio)) flags |=PM_FILE ;if ((flags &PM_PRESENT ) && -folio_precise_page_mapcount (folio, page) ==1 ) +__folio_page_mapped_exclusively (folio, page)) flags |=PM_MMAP_EXCLUSIVE ; }if (vma->vm_flags &VM_SOFTDIRTY )
@@ -1814,7 +1821,7 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end,pagemap_entry_t pme;if (folio && (flags &PM_PRESENT ) && -folio_precise_page_mapcount (folio, page + idx) ==1 ) +__folio_page_mapped_exclusively (folio, page)) cur_flags |=PM_MMAP_EXCLUSIVE ; pme =make_pme (frame, cur_flags);
Signature · 1 line
2.48.1
Re: [PATCH v2 17/20] fs/proc/task_mmu: remove per-page mapcount dependency for PM_MMAP_EXCLUSIVE (CONFIG_NO_PAGE_MAPCOUNT)
Quoted 27 lines · David Hildenbrand
Let's implement an alternative when per-page mapcounts in large folios are no longer maintained -- soon with
CONFIG_NO_PAGE_MAPCOUNT.
PM_MMAP_EXCLUSIVEwill now be set iffolio_likely_mapped_shared()is true -- when the folio is considered "mapped shared", including when it once was "mapped shared" but no longer is, as documented.This might result in and under-indication of "exclusively mapped", which is considered better than over-indicating it: under-estimating the USS (Unique Set Size) is better than over-estimating it.
As an alternative, we could simply remove that flag with
CONFIG_NO_PAGE_MAPCOUNTcompletely, but there might be value to it. So, let's keep it like that and document the behavior.Signed-off-by David Hildenbrand <[email protected]>--- Documentation/admin-guide/mm/pagemap.rst | 9 +++++++++ fs/proc/task_mmu.c | 11 +++++++++-- 2 files changed, 18 insertions(+), 2 deletions(-)Documentation/admin-guide/mm/pagemap.rst +0
-0@@ -37,6 +37,15 @@ There are four components to pagemap: precisely which pages are mapped (or in swap) and comparing mapped
Documentation/admin-guide/mm/pagemap.rst There are four components to pagemap: · line 47
Quoted 10 lines
Documentation/admin-guide/mm/pagemap.rst +8
-0pages between processes. + Note that in some kernel configurations, all pages part of a larger + allocation (e.g., THP) might be considered "mapped shared" if the large + allocation is considered "mapped shared": if not all pages are exclusive to + the same process. Further, some kernel configurations might consider larger + allocations "mapped shared", if they were at one point considered + "mapped shared", even if they would now be considered "exclusively mapped". + Consequently, in these kernel configurations, bit 56 might be set although + the page is actually "exclusively mapped"
I rewrote this yet another time to maybe make it clearer ...
+ Traditionally, bit 56 indicates that a page is mapped exactly once and bit + 56 is clear when a page is mapped multiple times, even when mapped in the + same process multiple times. In some kernel configurations, the semantics + for pages part of a larger allocation (e.g., THP) differ: bit 56 is set if + all pages part of the corresponding large allocation are *certainly* mapped + in the same process, even if the page is mapped multiple times in that + process. Bit 56 is clear when any page page of the larger allocation + is *maybe* mapped in a different process. In some cases, a large allocation + might be treated as "maybe mapped by multiple processes" even though this + is no longer the case.
(talking about "process" is not completely correct, it's actually "MMs"; but that might add more confusion here)
Signature · 3 lines
Cheers, David / dhildenb
PATCH v2 18/20 fs/proc/task_mmu: remove per-page mapcount dependency for "mapmax" (CONFIG_NO_PAGE_MAPCOUNT)
In reply to David HildenbrandLet's implement an alternative when per-page mapcounts in large folios are no longer maintained -- soon with CONFIG_NO_PAGE_MAPCOUNT.
For calculating "mapmax", we now use the average per-page mapcount in a large folio instead of the per-page mapcount.
For hugetlb folios and folios that are not partially mapped into MMs, there is no change.
Likely, this change will not matter much in practice, and an alternative might be to simple remove this stat with CONFIG_NO_PAGE_MAPCOUNT. However, there might be value to it, so let's keep it like that and document the behavior.
--- Documentation/filesystems/proc.rst | 5 +++++ fs/proc/task_mmu.c | 7 ++++++- 2 files changed, 11 insertions(+), 1 deletion(-)
Documentation/filesystems/proc.rst +5 -0
@@ -686,6 +686,11 @@ Where: node locality page counters (N0 == node0, N1 == node1, ...) and the kernel page size, in KB, that is backing the mapping up. +Note that some kernel configurations do not track the precise number of times +a page part of a larger allocation (e.g., THP) is mapped. In these +configurations, "mapmax" might corresponds to the average number of mappings +per page in such a larger allocation instead. + 1.2 Kernel data ---------------
fs/proc/task_mmu.c +6 -1
@@ -2862,7 +2862,12 @@ static void gather_stats(struct page *page, struct numa_maps *md, int pte_dirty, unsigned long nr_pages) {struct folio *folio =page_folio (page); -int count =folio_precise_page_mapcount (folio, page); +int count; + +if (IS_ENABLED (CONFIG_PAGE_MAPCOUNT )) + count =folio_precise_page_mapcount (folio, page); +else + count =min_t (int,folio_average_page_mapcount (folio),1 ); md->pages += nr_pages;if (pte_dirty ||folio_test_dirty (folio))
Signature · 1 line
2.48.1
Zi Yan
reviewerRe: [PATCH v2 18/20] fs/proc/task_mmu: remove per-page mapcount dependency for "mapmax" (CONFIG_NO_PAGE_MAPCOUNT)
Quoted 41 lines · David Hildenbrand
Let's implement an alternative when per-page mapcounts in large folios are no longer maintained -- soon with
CONFIG_NO_PAGE_MAPCOUNT.For calculating "mapmax", we now use the average per-page mapcount in a large folio instead of the per-page mapcount.
For hugetlb folios and folios that are not partially mapped into MMs, there is no change.
Likely, this change will not matter much in practice, and an alternative might be to simple remove this stat with
CONFIG_NO_PAGE_MAPCOUNT. However, there might be value to it, so let's keep it like that and document the behavior.Signed-off-by David Hildenbrand <[email protected]>--- Documentation/filesystems/proc.rst | 5 +++++ fs/proc/task_mmu.c | 7 ++++++- 2 files changed, 11 insertions(+), 1 deletion(-)Documentation/filesystems/proc.rst +5
-0@@ -686,6 +686,11 @@ Where: node locality page counters (N0 == node0, N1 == node1, ...) and the kernel page size, in KB, that is backing the mapping up. +Note that some kernel configurations do not track the precise number of times +a page part of a larger allocation (e.g., THP) is mapped. In these +configurations, "mapmax" might corresponds to the average number of mappings +per page in such a larger allocation instead. + 1.2 Kernel data ---------------fs/proc/task_mmu.c +0
-0
fs/proc/task_mmu.c static void gather_stats(struct page *page, struct numa_maps *md, int pte_dirty, · line 2870
Quoted 10 lines
fs/proc/task_mmu.c +6
-1@@ -2862,7 +2862,12 @@ static void gather_stats(struct page *page, struct numa_maps *md, int pte_dirty, unsigned long nr_pages) {struct folio *folio =page_folio (page); -int count =folio_precise_page_mapcount (folio, page); +int count; + +if (IS_ENABLED (CONFIG_PAGE_MAPCOUNT )) + count =folio_precise_page_mapcount (folio, page); +else + count =min_t (int,folio_average_page_mapcount (folio),1 );
s/min/max ?
Otherwise, count is at most 1. Anyway, if you change folio_average_page_mapcount() as I indicated in patch 16, this will become count = folio_average_page_mapcount(folio).
Quoted 3 lines
fs/proc/task_mmu.c +0
-0md->pages += nr_pages;if (pte_dirty ||folio_test_dirty (folio))
Signature · 2 lines
Best Regards, Yan, Zi
Re: [PATCH v2 18/20] fs/proc/task_mmu: remove per-page mapcount dependency for "mapmax" (CONFIG_NO_PAGE_MAPCOUNT)
Quoted 46 lines · Zi Yan
Quoted 45 lines · David Hildenbrand
Let's implement an alternative when per-page mapcounts in large folios are no longer maintained -- soon with
CONFIG_NO_PAGE_MAPCOUNT.For calculating "mapmax", we now use the average per-page mapcount in a large folio instead of the per-page mapcount.
For hugetlb folios and folios that are not partially mapped into MMs, there is no change.
Likely, this change will not matter much in practice, and an alternative might be to simple remove this stat with
CONFIG_NO_PAGE_MAPCOUNT. However, there might be value to it, so let's keep it like that and document the behavior.Signed-off-by David Hildenbrand <[email protected]>--- Documentation/filesystems/proc.rst | 5 +++++ fs/proc/task_mmu.c | 7 ++++++- 2 files changed, 11 insertions(+), 1 deletion(-)Documentation/filesystems/proc.rst +5
-0@@ -686,6 +686,11 @@ Where: node locality page counters (N0 == node0, N1 == node1, ...) and the kernel page size, in KB, that is backing the mapping up. +Note that some kernel configurations do not track the precise number of times +a page part of a larger allocation (e.g., THP) is mapped. In these +configurations, "mapmax" might corresponds to the average number of mappings +per page in such a larger allocation instead. + 1.2 Kernel data ---------------fs/proc/task_mmu.c +0
-1@@ -2862,7 +2862,12 @@ static void gather_stats(struct page *page, struct numa_maps *md, int pte_dirty, unsigned long nr_pages) {struct folio *folio =page_folio (page); -int count =folio_precise_page_mapcount (folio, page);
Quoted 8 lines
fs/proc/task_mmu.c static void gather_stats(struct page *page, struct numa_maps *md, int pte_dirty, · line 2870
Quoted 6 lines
fs/proc/task_mmu.c +6
-0+int count; + +if (IS_ENABLED (CONFIG_PAGE_MAPCOUNT )) + count =folio_precise_page_mapcount (folio, page); +else + count =min_t (int,folio_average_page_mapcount (folio),1 );s/min/max ?
Indeed, thanks!
Quoted 4 lines
Otherwise, count is at most 1. Anyway, if you change
folio_average_page_mapcount()as I indicated in patch 16, this will become count =folio_average_page_mapcount(folio).
No, the average should not be 1 just because a single subpage is mapped.
Signature · 3 lines
Cheers, David / dhildenb
PATCH v2 19/20 fs/proc/task_mmu: remove per-page mapcount dependency for smaps/smaps_rollup (CONFIG_NO_PAGE_MAPCOUNT)
In reply to David HildenbrandLet's implement an alternative when per-page mapcounts in large folios are no longer maintained -- soon with CONFIG_NO_PAGE_MAPCOUNT.
When computing the output for smaps / smaps_rollups, in particular when calculating the USS (Unique Set Size) and the PSS (Proportional Set Size), we still rely on per-page mapcounts.
To determine private vs. shared, we'll use folio_likely_mapped_shared(), similar to how we handle PM_MMAP_EXCLUSIVE. Similarly, we might now under-estimate the USS and count pages towards "shared" that are actually "private" ("exclusively mapped").
When calculating the PSS, we'll now also use the average per-page mapcount for large folios: this can result in both, an over-estimation and an under-estimation of the PSS. The difference is not expected to matter much in practice, but we'll have to learn as we go.
We can now provide folio_precise_page_mapcount() only with CONFIG_PAGE_MAPCOUNT, and remove one of the last users of per-page mapcounts when CONFIG_NO_PAGE_MAPCOUNT is enabled.
Document the new behavior.
--- Documentation/filesystems/proc.rst | 13 +++++++++++++ fs/proc/internal.h | 8 ++++++++ fs/proc/task_mmu.c | 17 +++++++++++++++-- 3 files changed, 36 insertions(+), 2 deletions(-)
Documentation/filesystems/proc.rst +13 -0
@@ -506,6 +506,19 @@ Note that even a page which is part of a MAP_SHARED mapping, but has only a single pte mapped, i.e. is currently used by only one process, is accounted as private and not as shared. +Note that in some kernel configurations, all pages part of a larger allocation +(e.g., THP) might be considered "shared" if the large allocation is +considered "shared": if not all pages are exclusive to the same process. +Further, some kernel configurations might consider larger allocations "shared", +if they were at one point considered "shared", even if they would now be +considered "exclusive". + +Some kernel configurations do not track the precise number of times a page part +of a larger allocation is mapped. In this case, when calculating the PSS, the +average number of mappings per page in this larger allocation might be used +as an approximation for the number of mappings of a page. The PSS calculation +will be imprecise in this case. + "Referenced" indicates the amount of memory currently marked as referenced or accessed.
fs/proc/internal.h +8 -0
@@ -143,6 +143,7 @@ unsigned name_to_int(const struct qstr *qstr);/* Worst case buffer size needed for holding an integer. */ #define PROC_NUMBUF 13 +#ifdef CONFIG_PAGE_MAPCOUNT /** * folio_precise_page_mapcount() - Number of mappings of this folio page. * @folio: The folio.
@@ -173,6 +174,13 @@ static inline int folio_precise_page_mapcount(struct folio *folio, return mapcount; } +#else /* !CONFIG_PAGE_MAPCOUNT */ +static inline int folio_precise_page_mapcount (struct folio *folio, +struct page *page) +{ +BUILD_BUG (); +} +#endif /* CONFIG_PAGE_MAPCOUNT */ /** * folio_average_page_mapcount() - Average number of mappings per page in this
fs/proc/task_mmu.c +15 -2
@@ -707,6 +707,8 @@ static void smaps_account(struct mem_size_stats *mss, struct page *page,struct folio *folio =page_folio (page);int i, nr = compound ?compound_nr (page) :1 ;unsigned long size = nr *PAGE_SIZE ; +bool exclusive; +int mapcount;/* * First accumulate quantities that depend only on |size| and the type
@@ -747,18 +749,29 @@ static void smaps_account(struct mem_size_stats *mss, struct page *page, dirty, locked, present);return ; } + +if (IS_ENABLED (CONFIG_NO_PAGE_MAPCOUNT )) { + mapcount =folio_average_page_mapcount (folio); + exclusive = !folio_maybe_mapped_shared (folio); + } +/* * We obtain a snapshot of the mapcount. Without holding the folio lock * this snapshot can be slightly wrong as we cannot always read the * mapcount atomically. */ for (i =0 ; i < nr; i++, page++) { -int mapcount =folio_precise_page_mapcount (folio, page);unsigned long pss =PAGE_SIZE <<PSS_SHIFT ; + +if (IS_ENABLED (CONFIG_PAGE_MAPCOUNT )) { + mapcount =folio_precise_page_mapcount (folio, page); + exclusive = mapcount <2 ; + } +if (mapcount >=2 ) pss /= mapcount;smaps_page_accumulate (mss, folio,PAGE_SIZE , pss, - dirty, locked, mapcount <2 ); + dirty, locked, exclusive); } }
Signature · 1 line
2.48.1
Zi Yan
reviewerRe: [PATCH v2 19/20] fs/proc/task_mmu: remove per-page mapcount dependency for smaps/smaps_rollup (CONFIG_NO_PAGE_MAPCOUNT)
Quoted 94 lines · David Hildenbrand
Let's implement an alternative when per-page mapcounts in large folios are no longer maintained -- soon with
CONFIG_NO_PAGE_MAPCOUNT.When computing the output for smaps /
smaps_rollups, in particular when calculating the USS (Unique Set Size) and the PSS (Proportional Set Size), we still rely on per-page mapcounts.To determine private vs. shared, we'll use
folio_likely_mapped_shared(), similar to how we handlePM_MMAP_EXCLUSIVE. Similarly, we might now under-estimate the USS and count pages towards "shared" that are actually "private" ("exclusively mapped").When calculating the PSS, we'll now also use the average per-page mapcount for large folios: this can result in both, an over-estimation and an under-estimation of the PSS. The difference is not expected to matter much in practice, but we'll have to learn as we go.
We can now provide
folio_precise_page_mapcount()only withCONFIG_PAGE_MAPCOUNT, and remove one of the last users of per-page mapcounts whenCONFIG_NO_PAGE_MAPCOUNTis enabled.Document the new behavior.
Signed-off-by David Hildenbrand <[email protected]>--- Documentation/filesystems/proc.rst | 13 +++++++++++++ fs/proc/internal.h | 8 ++++++++ fs/proc/task_mmu.c | 17 +++++++++++++++-- 3 files changed, 36 insertions(+), 2 deletions(-)Documentation/filesystems/proc.rst +13
-0@@ -506,6 +506,19 @@ Note that even a page which is part of a MAP_SHARED mapping, but has only a single pte mapped, i.e. is currently used by only one process, is accounted as private and not as shared. +Note that in some kernel configurations, all pages part of a larger allocation +(e.g., THP) might be considered "shared" if the large allocation is +considered "shared": if not all pages are exclusive to the same process. +Further, some kernel configurations might consider larger allocations "shared", +if they were at one point considered "shared", even if they would now be +considered "exclusive". + +Some kernel configurations do not track the precise number of times a page part +of a larger allocation is mapped. In this case, when calculating the PSS, the +average number of mappings per page in this larger allocation might be used +as an approximation for the number of mappings of a page. The PSS calculation +will be imprecise in this case. + "Referenced" indicates the amount of memory currently marked as referenced or accessed.fs/proc/internal.h +8
-0@@ -143,6 +143,7 @@ unsigned name_to_int(const struct qstr *qstr);/* Worst case buffer size needed for holding an integer. */ #define PROC_NUMBUF 13 +#ifdef CONFIG_PAGE_MAPCOUNT /** * folio_precise_page_mapcount() - Number of mappings of this folio page. * @folio: The folio. @@ -173,6 +174,13 @@ static inline int folio_precise_page_mapcount(struct folio *folio, return mapcount; } +#else /* !CONFIG_PAGE_MAPCOUNT */ +static inline int folio_precise_page_mapcount (struct folio *folio, +struct page *page) +{ +BUILD_BUG (); +} +#endif /* CONFIG_PAGE_MAPCOUNT */ /** * folio_average_page_mapcount() - Average number of mappings per page in this fs/proc/task_mmu.c +2
-0@@ -707,6 +707,8 @@ static void smaps_account(struct mem_size_stats *mss, struct page *page,struct folio *folio =page_folio (page);int i, nr = compound ?compound_nr (page) :1 ;unsigned long size = nr *PAGE_SIZE ; +bool exclusive; +int mapcount;/* * First accumulate quantities that depend only on |size| and the type
fs/proc/task_mmu.c static void smaps_account(struct mem_size_stats *mss, struct page *page, · line 754
Quoted 6 lines
fs/proc/task_mmu.c +3
-0@@ -747,18 +749,29 @@ static void smaps_account(struct mem_size_stats *mss, struct page *page, dirty, locked, present);return ; } + +if (IS_ENABLED (CONFIG_NO_PAGE_MAPCOUNT )) { + mapcount =folio_average_page_mapcount (folio);
This seems inconsistent with how folio_average_page_mapcount() is used in patch 16 and 18.
Quoted 25 lines
fs/proc/task_mmu.c +10
-2+ exclusive = !folio_maybe_mapped_shared (folio); + } +/* * We obtain a snapshot of the mapcount. Without holding the folio lock * this snapshot can be slightly wrong as we cannot always read the * mapcount atomically. */ for (i =0 ; i < nr; i++, page++) { -int mapcount =folio_precise_page_mapcount (folio, page);unsigned long pss =PAGE_SIZE <<PSS_SHIFT ; + +if (IS_ENABLED (CONFIG_PAGE_MAPCOUNT )) { + mapcount =folio_precise_page_mapcount (folio, page); + exclusive = mapcount <2 ; + } +if (mapcount >=2 ) pss /= mapcount;smaps_page_accumulate (mss, folio,PAGE_SIZE , pss, - dirty, locked, mapcount <2 ); + dirty, locked, exclusive); } }
Signature · 2 lines
Best Regards, Yan, Zi
Re: [PATCH v2 19/20] fs/proc/task_mmu: remove per-page mapcount dependency for smaps/smaps_rollup (CONFIG_NO_PAGE_MAPCOUNT)
Quoted 92 lines · Zi Yan
Quoted 91 lines · David Hildenbrand
Let's implement an alternative when per-page mapcounts in large folios are no longer maintained -- soon with
CONFIG_NO_PAGE_MAPCOUNT.When computing the output for smaps /
smaps_rollups, in particular when calculating the USS (Unique Set Size) and the PSS (Proportional Set Size), we still rely on per-page mapcounts.To determine private vs. shared, we'll use
folio_likely_mapped_shared(), similar to how we handlePM_MMAP_EXCLUSIVE. Similarly, we might now under-estimate the USS and count pages towards "shared" that are actually "private" ("exclusively mapped").When calculating the PSS, we'll now also use the average per-page mapcount for large folios: this can result in both, an over-estimation and an under-estimation of the PSS. The difference is not expected to matter much in practice, but we'll have to learn as we go.
We can now provide
folio_precise_page_mapcount()only withCONFIG_PAGE_MAPCOUNT, and remove one of the last users of per-page mapcounts whenCONFIG_NO_PAGE_MAPCOUNTis enabled.Document the new behavior.
Signed-off-by David Hildenbrand <[email protected]>--- Documentation/filesystems/proc.rst | 13 +++++++++++++ fs/proc/internal.h | 8 ++++++++ fs/proc/task_mmu.c | 17 +++++++++++++++-- 3 files changed, 36 insertions(+), 2 deletions(-)Documentation/filesystems/proc.rst +13
-0@@ -506,6 +506,19 @@ Note that even a page which is part of a MAP_SHARED mapping, but has only a single pte mapped, i.e. is currently used by only one process, is accounted as private and not as shared. +Note that in some kernel configurations, all pages part of a larger allocation +(e.g., THP) might be considered "shared" if the large allocation is +considered "shared": if not all pages are exclusive to the same process. +Further, some kernel configurations might consider larger allocations "shared", +if they were at one point considered "shared", even if they would now be +considered "exclusive". + +Some kernel configurations do not track the precise number of times a page part +of a larger allocation is mapped. In this case, when calculating the PSS, the +average number of mappings per page in this larger allocation might be used +as an approximation for the number of mappings of a page. The PSS calculation +will be imprecise in this case. + "Referenced" indicates the amount of memory currently marked as referenced or accessed.fs/proc/internal.h +8
-0@@ -143,6 +143,7 @@ unsigned name_to_int(const struct qstr *qstr);/* Worst case buffer size needed for holding an integer. */ #define PROC_NUMBUF 13 +#ifdef CONFIG_PAGE_MAPCOUNT /** * folio_precise_page_mapcount() - Number of mappings of this folio page. * @folio: The folio. @@ -173,6 +174,13 @@ static inline int folio_precise_page_mapcount(struct folio *folio, return mapcount; } +#else /* !CONFIG_PAGE_MAPCOUNT */ +static inline int folio_precise_page_mapcount (struct folio *folio, +struct page *page) +{ +BUILD_BUG (); +} +#endif /* CONFIG_PAGE_MAPCOUNT */ /** * folio_average_page_mapcount() - Average number of mappings per page in this fs/proc/task_mmu.c +2
-0@@ -707,6 +707,8 @@ static void smaps_account(struct mem_size_stats *mss, struct page *page,struct folio *folio =page_folio (page);int i, nr = compound ?compound_nr (page) :1 ;unsigned long size = nr *PAGE_SIZE ; +bool exclusive; +int mapcount;
Quoted 12 lines
fs/proc/task_mmu.c static void smaps_account(struct mem_size_stats *mss, struct page *page, · line 754
Quoted 9 lines
fs/proc/task_mmu.c +3
-0/* * First accumulate quantities that depend only on |size| and the type @@ -747,18 +749,29 @@ static void smaps_account(struct mem_size_stats *mss, struct page *page, dirty, locked, present);return ; } + +if (IS_ENABLED (CONFIG_NO_PAGE_MAPCOUNT )) { + mapcount =folio_average_page_mapcount (folio);This seems inconsistent with how
folio_average_page_mapcount()is used in patch 16 and 18.
It only looks that way. Note that the code below only does
Quoted 2 lines
fs/proc/task_mmu.c static void smaps_account(struct mem_size_stats *mss, struct page *page, · line 772
Quoted 2 lines
fs/proc/task_mmu.c +0
-0if (mapcount >=2 ) pss /= mapcount;
Having it as 0 or 1 doesn't matter in that regard.
Thanks!
Signature · 3 lines
Cheers, David / dhildenb
PATCH v2 20/20 mm: stop maintaining the per-page mapcount of large folios (CONFIG_NO_PAGE_MAPCOUNT)
In reply to David HildenbrandEverything is in place to stop using the per-page mapcounts in large
folios: the mapcount of tail pages will always be logically 0 (-1 value), just like it currently is for hugetlb folios already, and the page mapcount of the head page is either 0 (-1 value) or contains a page type (e.g., hugetlb).
Maintaining _nr_pages_mapped without per-page mapcounts is impossible, so that one also has to go with CONFIG_NO_PAGE_MAPCOUNT.
There are two remaining implications:
(1) Per-node, per-cgroup and per-lruvec stats of "NR_ANON_MAPPED" ("mapped anonymous memory") and "NR_FILE_MAPPED" ("mapped file memory"):
As soon as any page of the folio is mapped -- folio_mapped() -- we now account the complete folio as mapped. Once the last page is unmapped -- !folio_mapped() -- we account the complete folio as unmapped.
This implies that ...
* "AnonPages" and "Mapped" in /proc/meminfo and /sys/devices/system/node/*/meminfo
* cgroup v2: "anon" and "file_mapped" in "memory.stat" and "memory.numa_stat"
* cgroup v1: "rss" and "mapped_file" in "memory.stat" and "memory.numa_stat
... can now appear higher than before. But note that these folios do consume that memory, simply not all pages are actually currently mapped.
It's worth nothing that other accounting in the kernel (esp. cgroup charging on allocation) is not affected by this change.
[why oh why is "anon" called "rss" in cgroup v1]
(2) Detecting partial mappings
Detecting whether anon THPs are partially mapped gets a bit more unreliable. As long as a single MM maps such a large folio ("exclusively mapped"), we can reliably detect it. Especially before fork() / after a short-lived child process quit, we will detect partial mappings reliably, which is the common case.
In essence, if the average per-page mapcount in an anon THP is < 1, we know for sure that we have a partial mapping.
However, as soon as multiple MMs are involved, we might miss detecting partial mappings: this might be relevant with long-lived child processes. If we have a fully-mapped anon folio before fork(), once our child processes and our parent all unmap (zap/COW) the same pages (but not the complete folio), we might not detect the partial mapping. However, once the child processes quit we would detect the partial mapping.
How relevant this case is in practice remains to be seen. Swapout/migration will likely mitigate this.
In the future, RMAP walkers could check for that for that case (e.g., when collecting access bits during reclaim) and simply flag them for deferred-splitting.
--- .../admin-guide/cgroup-v1/memory.rst | 4 + Documentation/admin-guide/cgroup-v2.rst | 10 ++- Documentation/filesystems/proc.rst | 10 ++- Documentation/mm/transhuge.rst | 31 +++++-- include/linux/rmap.h | 35 ++++++-- mm/internal.h | 5 +- mm/page_alloc.c | 3 +- mm/rmap.c | 80 +++++++++++++++++-- 8 files changed, 150 insertions(+), 28 deletions(-)
Documentation/admin-guide/cgroup-v1/memory.rst +4 -0
@@ -609,6 +609,10 @@ memory.stat file includes following statistics: 'rss + mapped_file" will give you resident set size of cgroup. + Note that some kernel configurations might account complete larger + allocations (e.g., THP) towards 'rss' and 'mapped_file', even if + only some, but not all that memory is mapped. + (Note: file and shmem may be shared among other cgroups. In that case, mapped_file is accounted only when the memory cgroup is owner of page cache.)
Documentation/admin-guide/cgroup-v2.rst +8 -2
@@ -1448,7 +1448,10 @@ The following nested keys are defined. anon Amount of memory used in anonymous mappings such as - brk(), sbrk(), and mmap(MAP_ANONYMOUS) + brk(), sbrk(), and mmap(MAP_ANONYMOUS). Note that + some kernel configurations might account complete larger + allocations (e.g., THP) if only some, but not all the + memory of such an allocation is mapped anymore. file Amount of memory used to cache filesystem data,
@@ -1491,7 +1494,10 @@ The following nested keys are defined. Amount of application memory swapped out to zswap. file_mapped - Amount of cached filesystem data mapped with mmap() + Amount of cached filesystem data mapped with mmap(). Note + that some kernel configurations might account complete + larger allocations (e.g., THP) if only some, but not + not all the memory of such an allocation is mapped. file_dirty Amount of cached filesystem data that was modified but
Documentation/filesystems/proc.rst +8 -2
@@ -1150,9 +1150,15 @@ Dirty Writeback Memory which is actively being written back to the disk AnonPages - Non-file backed pages mapped into userspace page tables + Non-file backed pages mapped into userspace page tables. Note that + some kernel configurations might consider all pages part of a + larger allocation (e.g., THP) as "mapped", as soon as a single + page is mapped. Mapped - files which have been mmapped, such as libraries + files which have been mmapped, such as libraries. Note that some + kernel configurations might consider all pages part of a larger + allocation (e.g., THP) as "mapped", as soon as a single page is + mapped. Shmem Total memory used by shared memory (shmem) and tmpfs KReclaimable
Documentation/mm/transhuge.rst +23 -8
@@ -116,23 +116,28 @@ pages: succeeds on tail pages. - map/unmap of a PMD entry for the whole THP increment/decrement - folio->_entire_mapcount, increment/decrement folio->_large_mapcount - and also increment/decrement folio->_nr_pages_mapped by ENTIRELY_MAPPED - when _entire_mapcount goes from -1 to 0 or 0 to -1. + folio->_entire_mapcount and folio->_large_mapcount. We also maintain the two slots for tracking MM owners (MM ID and corresponding mapcount), and the current status ("maybe mapped shared" vs. "mapped exclusively"). + With CONFIG_PAGE_MAPCOUNT, we also increment/decrement + folio->_nr_pages_mapped by ENTIRELY_MAPPED when _entire_mapcount goes + from -1 to 0 or 0 to -1. + - map/unmap of individual pages with PTE entry increment/decrement - page->_mapcount, increment/decrement folio->_large_mapcount and also - increment/decrement folio->_nr_pages_mapped when page->_mapcount goes - from -1 to 0 or 0 to -1 as this counts the number of pages mapped by PTE. + folio->_large_mapcount. We also maintain the two slots for tracking MM owners (MM ID and corresponding mapcount), and the current status ("maybe mapped shared" vs. "mapped exclusively"). + With CONFIG_PAGE_MAPCOUNT, we also increment/decrement + page->_mapcount and increment/decrement folio->_nr_pages_mapped when + page->_mapcount goes from -1 to 0 or 0 to -1 as this counts the number + of pages mapped by PTE. + split_huge_page internally has to distribute the refcounts in the head page to the tail pages before clearing all PG_head/tail bits from the page structures. It can be done easily for refcounts taken by page table
@@ -159,8 +164,8 @@ clear where references should go after split: it will stay on the head page. Note that split_huge_pmd() doesn't have any limitations on refcounting: pmd can be split at any point and never fails. -Partial unmap and deferred_split_folio() -======================================== +Partial unmap and deferred_split_folio() (anon THP only) +======================================================== Unmapping part of THP (with munmap() or other way) is not going to free memory immediately. Instead, we detect that a subpage of THP is not in use
@@ -175,3 +180,13 @@ a THP crosses a VMA boundary. The function deferred_split_folio() is used to queue a folio for splitting. The splitting itself will happen when we get memory pressure via shrinker interface. + +With CONFIG_PAGE_MAPCOUNT, we reliably detect partial mappings based on +folio->_nr_pages_mapped. + +With CONFIG_NO_PAGE_MAPCOUNT, we detect partial mappings based on the +average per-page mapcount in a THP: if the average is < 1, an anon THP is +certainly partially mapped. As long as only a single process maps a THP, +this detection is reliable. With long-running child processes, there can +be scenarios where partial mappings can currently not be detected, and +might need asynchronous detection during memory reclaim in the future.
include/linux/rmap.h +29 -6
@@ -240,7 +240,7 @@ static __always_inline void folio_set_large_mapcount(struct folio *folio,folio_set_mm_id (folio,0 , vma->vm_mm->mm_id); } -static __always_inline void folio_add_large_mapcount(struct folio *folio, +static __always_inline int folio_add_return_large_mapcount(struct folio *folio,int diff,struct vm_area_struct *vma) {const mm_id_t mm_id = vma->vm_mm->mm_id;
@@ -286,9 +286,11 @@ static __always_inline void folio_add_large_mapcount(struct folio *folio, folio->_mm_ids |=FOLIO_MM_IDS_SHARED_BIT ; }folio_unlock_large_mapcount (folio); +return new_mapcount_val +1 ; } +#define folio_add_large_mapcount folio_add_return_large_mapcount -static __always_inline void folio_sub_large_mapcount(struct folio *folio, +static __always_inline int folio_sub_return_large_mapcount(struct folio *folio,int diff,struct vm_area_struct *vma) {const mm_id_t mm_id = vma->vm_mm->mm_id;
@@ -331,7 +333,9 @@ static __always_inline void folio_sub_large_mapcount(struct folio *folio, folio->_mm_ids &= ~FOLIO_MM_IDS_SHARED_BIT ; out:folio_unlock_large_mapcount (folio); + return new_mapcount_val + 1; } +#define folio_sub_large_mapcount folio_sub_return_large_mapcount#else /* !CONFIG_MM_ID */ /* * See __folio_rmap_sanity_checks(), we might map large folios even without
@@ -350,17 +354,33 @@ static inline void folio_add_large_mapcount(struct folio *folio, atomic_add(diff, &folio->_large_mapcount); } +static inline int folio_add_return_large_mapcount (struct folio *folio, +int diff,struct vm_area_struct *vma) +{ +BUILD_BUG (); +} +static inline void folio_sub_large_mapcount (struct folio *folio,int diff,struct vm_area_struct *vma) {atomic_sub (diff, &folio->_large_mapcount); } + +static inline int folio_sub_return_large_mapcount (struct folio *folio, +int diff,struct vm_area_struct *vma) +{ +BUILD_BUG (); +}#endif /* CONFIG_MM_ID */ #define folio_inc_large_mapcount (folio, vma) \ folio_add_large_mapcount(folio, 1, vma) +#define folio_inc_return_large_mapcount (folio, vma) \ + folio_add_return_large_mapcount(folio, 1, vma)#define folio_dec_large_mapcount (folio, vma) \ folio_sub_large_mapcount(folio, 1, vma) +#define folio_dec_return_large_mapcount (folio, vma) \ + folio_sub_return_large_mapcount(folio, 1, vma)/* RMAP flags, currently only relevant for some anon rmap operations. */ typedef int __bitwise rmap_t ;
@@ -538,9 +558,11 @@ static __always_inline void __folio_dup_file_rmap(struct folio *folio,break ; } -do { -atomic_inc (&page->_mapcount); - }while (page++, --nr_pages >0 ); +if (IS_ENABLED (CONFIG_PAGE_MAPCOUNT )) { +do { +atomic_inc (&page->_mapcount); + }while (page++, --nr_pages >0 ); + }folio_add_large_mapcount (folio, orig_nr_pages, dst_vma);break ;case RMAP_LEVEL_PMD :
@@ -638,7 +660,8 @@ static __always_inline int __folio_try_dup_anon_rmap(struct folio *folio,do {if (PageAnonExclusive (page))ClearPageAnonExclusive (page); -atomic_inc (&page->_mapcount); +if (IS_ENABLED (CONFIG_PAGE_MAPCOUNT )) +atomic_inc (&page->_mapcount); }while (page++, --nr_pages >0 );folio_add_large_mapcount (folio, orig_nr_pages, dst_vma);break ;
mm/internal.h +4 -1
@@ -84,6 +84,8 @@ void page_writeback_init(void);*/ static inline int folio_nr_pages_mapped (const struct folio *folio) { +if (IS_ENABLED (CONFIG_NO_PAGE_MAPCOUNT )) +return -1 ;return atomic_read (&folio->_nr_pages_mapped) &FOLIO_PAGES_MAPPED ; }
@@ -721,7 +723,8 @@ static inline void prep_compound_head(struct page *page, unsigned int order)folio_set_order (folio, order);atomic_set (&folio->_large_mapcount,-1 ); -atomic_set (&folio->_nr_pages_mapped,0 ); +if (IS_ENABLED (CONFIG_PAGE_MAPCOUNT )) +atomic_set (&folio->_nr_pages_mapped,0 );if (IS_ENABLED (CONFIG_MM_ID )) { folio->_mm_ids =0 ; folio->_mm_id_mapcount[0 ] =-1 ;
mm/page_alloc.c +2 -1
@@ -951,7 +951,8 @@ static int free_tail_page_prepare(struct page *head_page, struct page *page)bad_page (page,"nonzero large_mapcount" ); goto out; } -if (unlikely (atomic_read (&folio->_nr_pages_mapped))) { +if (IS_ENABLED (CONFIG_PAGE_MAPCOUNT ) && +unlikely (atomic_read (&folio->_nr_pages_mapped))) {bad_page (page,"nonzero nr_pages_mapped" ); goto out; }
mm/rmap.c +72 -8
129 lines not on this page · raw message
Signature · 1 line
2.48.1