Skip to contentSkip to conversation

folio mapcount

DISCUSSION Matthew Wilcox 4 participants 8 messages Dec 15, 2021 – Dec 16, 2021 Memory management

lore
Share & tools
lore
#0 lore raw

I've been trying to understand whether we can simplify the mapcount handling for folios from the current situation with THPs. Let me quote the commit message from 53f9263baba6:

Quoted 28 lines

mm: rework mapcount accounting to enable 4k mapping of THPs

We're going to allow mapping of individual 4k pages of THP compound. It means we need to track mapcount on per small page basis.

Straight-forward approach is to use ->_mapcount in all subpages to track how many time this subpage is mapped with PMDs or PTEs combined. But this is rather expensive: mapping or unmapping of a THP page with PMD would require HPAGE_PMD_NR atomic operations instead of single we have now.

The idea is to store separately how many times the page was mapped as whole -- compound_mapcount. This frees up ->_mapcount in subpages to track PTE mapcount.

We use the same approach as with compound page destructor and compound order to store compound_mapcount: use space in first tail page, ->mapping this time.

Any time we map/unmap whole compound page (THP or hugetlb) -- we increment/decrement compound_mapcount. When we map part of compound page with PTE we operate on ->_mapcount of the subpage.

page_mapcount() counts both: PTE and PMD mappings of the page.

Basically, we have mapcount for a subpage spread over two counters. It makes tricky to detect when last mapcount for a page goes away.

Quoted 8 lines

We introduced PageDoubleMap() for this. When we split THP PMD for the first time and there's other PMD mapping left we offset up ->_mapcount in all subpages by one and set PG_double_map on the compound page. These additional references go away with last compound_mapcount.

This approach provides a way to detect when last mapcount goes away on per small page basis without introducing new overhead for most common cases.

What breaks if we simply track any mapping (whether by PMD or PTE) as an increment to the head page (aka folio's) refcount?

Essentially, we make the head mapcount 'the number of VMAs which contain a reference to any page in this folio'. We can remove PageDoubleMap. The tail refcounts will all be 0. If it's useful, we could introduce a 'partial_mapcount' which would be <= mapcount (but I don't know if it's useful). Splitting a PMD would not change ->_mapcount. Splitting the folio already causes the folio to be unmapped, so page faults will naturally re-increment ->_mapcount of each subpage.

We might need some additional logic to treat a large folio (aka compound page) as a single unit; that is, when we fault on one page, we place entries for all pages in this folio (that fit ...) into the page tables, so that we only account it once, even if it's not compatible with using a PMD.

#1 lore raw
Quoted 37 lines · Matthew Wilcox

I've been trying to understand whether we can simplify the mapcount handling for folios from the current situation with THPs. Let me quote the commit message from 53f9263baba6:

Quoted 33 lines

mm: rework mapcount accounting to enable 4k mapping of THPs

We're going to allow mapping of individual 4k pages of THP compound. It means we need to track mapcount on per small page basis.

Straight-forward approach is to use ->_mapcount in all subpages to track how many time this subpage is mapped with PMDs or PTEs combined. But this is rather expensive: mapping or unmapping of a THP page with PMD would require HPAGE_PMD_NR atomic operations instead of single we have now.

The idea is to store separately how many times the page was mapped as whole -- compound_mapcount. This frees up ->_mapcount in subpages to track PTE mapcount.

We use the same approach as with compound page destructor and compound order to store compound_mapcount: use space in first tail page, ->mapping this time.

Any time we map/unmap whole compound page (THP or hugetlb) -- we increment/decrement compound_mapcount. When we map part of compound page with PTE we operate on ->_mapcount of the subpage.

page_mapcount() counts both: PTE and PMD mappings of the page.

Basically, we have mapcount for a subpage spread over two counters. It makes tricky to detect when last mapcount for a page goes away.

We introduced PageDoubleMap() for this. When we split THP PMD for the first time and there's other PMD mapping left we offset up ->_mapcount in all subpages by one and set PG_double_map on the compound page. These additional references go away with last compound_mapcount.

Quoted 6 lines
Quoted 3 lines

This approach provides a way to detect when last mapcount goes away on per small page basis without introducing new overhead for most common cases.

What breaks if we simply track any mapping (whether by PMD or PTE) as an increment to the head page (aka folio's) refcount?

The obvious answer is CoW: as discussed yesterday we need exact mapcount to know if the page can be re-used or has to be copied.

Consider the case when you have folio mapped as PMD and then split into PTE page table (like with mprotect()). You get WP page fault on a page that has mapcount == 512. How would you know if we can re-use the 4k?

Also we need to detect case when the last mapping of a 4k in the folio has gone to trigger deferred_split_huge_page() logic.

Quoted 2 lines

Essentially, we make the head mapcount 'the number of VMAs which contain a reference to any page in this folio'.

Okay, so you will have mapcount == 2 or 3 for mprotect case above, not
512. But it doesn't help with answering question if the page can be re-used. You would need to do rmap walk to get the answer.

Note also that VMA lifecycle is different from page lifecycle:
MADV_DONTNEED removes mapping, but leaves VMA intact. Who would decrement mapcount here?

Quoted 7 lines

We can remove PageDoubleMap. The tail refcounts will all be 0. If it's useful, we could introduce a 'partial_mapcount' which would be <= mapcount (but I don't know if it's useful). Splitting a PMD would not change ->_mapcount. Splitting the folio already causes the folio to be unmapped, so page faults will naturally re-increment ->_mapcount of each subpage.

Quoted 5 lines

We might need some additional logic to treat a large folio (aka compound page) as a single unit; that is, when we fault on one page, we place entries for all pages in this folio (that fit ...) into the page tables, so that we only account it once, even if it's not compatible with using a PMD.

I still don't see a way to simplify mapcount for THP. But I'm preconsived becasue I'm the author of the current scheme.

Please, prove me wrong. I want to be mistaken. :)

Signature · 1 line
 Kirill A. Shutemov
#2 lore raw
Quoted 45 lines · Kirill A. Shutemov
Quoted 37 lines · Matthew Wilcox

I've been trying to understand whether we can simplify the mapcount handling for folios from the current situation with THPs. Let me quote the commit message from 53f9263baba6:

Quoted 33 lines

mm: rework mapcount accounting to enable 4k mapping of THPs

We're going to allow mapping of individual 4k pages of THP compound. It means we need to track mapcount on per small page basis.

Straight-forward approach is to use ->_mapcount in all subpages to track how many time this subpage is mapped with PMDs or PTEs combined. But this is rather expensive: mapping or unmapping of a THP page with PMD would require HPAGE_PMD_NR atomic operations instead of single we have now.

The idea is to store separately how many times the page was mapped as whole -- compound_mapcount. This frees up ->_mapcount in subpages to track PTE mapcount.

We use the same approach as with compound page destructor and compound order to store compound_mapcount: use space in first tail page, ->mapping this time.

Any time we map/unmap whole compound page (THP or hugetlb) -- we increment/decrement compound_mapcount. When we map part of compound page with PTE we operate on ->_mapcount of the subpage.

page_mapcount() counts both: PTE and PMD mappings of the page.

Basically, we have mapcount for a subpage spread over two counters. It makes tricky to detect when last mapcount for a page goes away.

We introduced PageDoubleMap() for this. When we split THP PMD for the first time and there's other PMD mapping left we offset up ->_mapcount in all subpages by one and set PG_double_map on the compound page. These additional references go away with last compound_mapcount.

Quoted 6 lines
Quoted 3 lines

This approach provides a way to detect when last mapcount goes away on per small page basis without introducing new overhead for most common cases.

What breaks if we simply track any mapping (whether by PMD or PTE) as an increment to the head page (aka folio's) refcount?

Quoted 6 lines

The obvious answer is CoW: as discussed yesterday we need exact mapcount to know if the page can be re-used or has to be copied.

Consider the case when you have folio mapped as PMD and then split into PTE page table (like with mprotect()). You get WP page fault on a page that has mapcount == 512. How would you know if we can re-use the 4k?

I was trying to say the exact opposite of that ... fortunately I rephrased it below. The scenario I think you're describing here is:

p = mmap(x, 2MB, PROT_READ|PROT_WRITE, ...): THP allocated mprotect(p, 4KB, PROT_READ): THP split.

And in that case, I would say the THP now has mapcount of 2 because there are 2 VMAs mapping it.

Quoted 2 lines

Also we need to detect case when the last mapping of a 4k in the folio has gone to trigger deferred_split_huge_page() logic.

I think you're referring to this logic in rmap.c:

        if (TestClearPageDoubleMap(page)) {
                /*
                 * Subpages can be mapped with PTEs too. Check how many of
                 * them are still mapped.
                 */
                for (i = 0, nr = 0; i < thp_nr_pages(page); i++) {
                        if (atomic_add_negative(-1, &page[i]._mapcount))
                                nr++;
                }

                /*
                 * Queue the page for deferred split if at least one small
                 * page of the compound page is unmapped, but at least one
                 * small page is still mapped.
                 */
                if (nr && nr < thp_nr_pages(page))
                        deferred_split_huge_page(page);

The 'partial_mapcount' idea I mentioned below could help with this. Checking that they're identical might be racy, though.

Quoted 10 lines
Quoted 2 lines

Essentially, we make the head mapcount 'the number of VMAs which contain a reference to any page in this folio'.

Okay, so you will have mapcount == 2 or 3 for mprotect case above, not
512. But it doesn't help with answering question if the page can be re-used. You would need to do rmap walk to get the answer.

Note also that VMA lifecycle is different from page lifecycle:
MADV_DONTNEED removes mapping, but leaves VMA intact. Who would decrement mapcount here?

I think page_remove_rmap() does when called from zap_huge_pmd(). The tricky part is handling partial DONTNEED calls. eg we could have a 2MB page (if we're talking about shmem, it could even be mapped askew), MADV_DONTNEED the first 512KB, then MADV_DONTNEED the last 512KB, then finally MADV_DONTNEED the middle 1MB and only at the third call should the mapcount be decremented.

So a zap call has to check all the PTE/PMD entries in the range of the (folio intersect vma) to be sure that there are no more references to a part of this folio. It might not be terribly fast, but it probably won't be noticable compared to all the other costs of doing a munmap().

Quoted 13 lines
Quoted 7 lines

We can remove PageDoubleMap. The tail refcounts will all be 0. If it's useful, we could introduce a 'partial_mapcount' which would be <= mapcount (but I don't know if it's useful). Splitting a PMD would not change ->_mapcount. Splitting the folio already causes the folio to be unmapped, so page faults will naturally re-increment ->_mapcount of each subpage.

Quoted 5 lines

We might need some additional logic to treat a large folio (aka compound page) as a single unit; that is, when we fault on one page, we place entries for all pages in this folio (that fit ...) into the page tables, so that we only account it once, even if it's not compatible with using a PMD.

Quoted 4 lines

I still don't see a way to simplify mapcount for THP. But I'm preconsived becasue I'm the author of the current scheme.

Please, prove me wrong. I want to be mistaken. :)

I'm just trying to learn enough to make sensible suggestions for simplification. As yesterday's call proved, there are all kinds of corner cases when messing with mapcount and refcount.

#3 lore raw
Quoted 5 lines · Matthew Wilcox

p = mmap(x, 2MB, PROT_READ|PROT_WRITE, ...): THP allocated mprotect(p, 4KB, PROT_READ): THP split.

And in that case, I would say the THP now has mapcount of 2 because there are 2 VMAs mapping it.

At least today mapcount is only loosely connected to VMAs. It really counts the number of PUD/PTEs that point at parts of the memory.

If, under the PTL, you observe a mapcount of 1 then you know that the PUD/PTE you have under lock is the ONLY PUD/PTE that refers to this page and will remain so while the lock is held.

So, today the above ends up with a mapcount of 1 and when we take a COW fault we can re-use the page.

If the above ends up with a mapcount of 2 then COW will copy not re-use, which will cause unexpected data corruption in all those annoying side cases.

The actual value of mapcount doesn't matter, the main issue is if it is 1 or not, and if 1 the determination needs to be reliable and race free.

Putting the mapcount on the head page is sort of an optimization 'all tail pages share the same mapcount' and the doublemap thing is about exiting that optimization with minimal locking.

There is other stuff going on too that can possibly do other things but this seems to be the primary difficult task.

(IIRC anyhow, from when I looked at this with David a few months ago)

Quoted 3 lines

I'm just trying to learn enough to make sensible suggestions for simplification. As yesterday's call proved, there are all kinds of corner cases when messing with mapcount and refcount.

It would be amazing to get some better idea!

Jason

#4 lore raw
Quoted 9 lines · Jason Gunthorpe
Quoted 5 lines · Matthew Wilcox

p = mmap(x, 2MB, PROT_READ|PROT_WRITE, ...): THP allocated mprotect(p, 4KB, PROT_READ): THP split.

And in that case, I would say the THP now has mapcount of 2 because there are 2 VMAs mapping it.

At least today mapcount is only loosely connected to VMAs. It really counts the number of PUD/PTEs that point at parts of the memory.

Careful. Currently, you need to distinguish between total_mapcount(), page_trans_huge_mapcount() and page_mapcount(). Take a look at __page_mapcount() to be sure you really know what the mapcount "really" counts today ...

(also I'm going to assume that when you said PUD you really mean PMD throughout)

Quoted 10 lines

If, under the PTL, you observe a mapcount of 1 then you know that the PUD/PTE you have under lock is the ONLY PUD/PTE that refers to this page and will remain so while the lock is held.

So, today the above ends up with a mapcount of 1 and when we take a COW fault we can re-use the page.

If the above ends up with a mapcount of 2 then COW will copy not re-use, which will cause unexpected data corruption in all those annoying side cases.

As I understood David's presentation yesterday, we actually have data corruption issues in all the annoying side cases with THPs in current upstream, so that's no worse than we have now. But let's see if we can avoid them.

It feels like what we want from a COW perspective is a count of the number of MMs mapping a page, not the number of VMAs, PTEs or PMDs mapping the page. Right?

So here's a corner case ...

p = mmap(x, 2MB, PROT_READ|PROT_WRITE, ...): THP allocated mremap(p + 128K, 128K, 128K, MREMAP_MAYMOVE | MREMAP_FIXED, p + 2MB):
PMD split

Should mapcount be 1 or 2 at this point? Does the answer change if it's an anonymous or file page? (I think it might; perhaps if you do this to an anon THP, we split the anon THP and each page is now mapped once, whereas if you do it to a file page, the page cache does not split the page, but it does count the extra mapping. also the answer may be different between MAP_PRIVATE and MAP_SHARED)

Quoted 21 lines

The actual value of mapcount doesn't matter, the main issue is if it is 1 or not, and if 1 the determination needs to be reliable and race free.

Putting the mapcount on the head page is sort of an optimization 'all tail pages share the same mapcount' and the doublemap thing is about exiting that optimization with minimal locking.

There is other stuff going on too that can possibly do other things but this seems to be the primary difficult task.

(IIRC anyhow, from when I looked at this with David a few months ago)

Quoted 3 lines

I'm just trying to learn enough to make sensible suggestions for simplification. As yesterday's call proved, there are all kinds of corner cases when messing with mapcount and refcount.

It would be amazing to get some better idea!

Jason

#5 lore raw
Quoted 8 lines · Matthew Wilcox
Quoted 7 lines · Jason Gunthorpe
Quoted 5 lines · Matthew Wilcox

p = mmap(x, 2MB, PROT_READ|PROT_WRITE, ...): THP allocated mprotect(p, 4KB, PROT_READ): THP split.

And in that case, I would say the THP now has mapcount of 2 because there are 2 VMAs mapping it.

Quoted 7 lines
Quoted 2 lines

At least today mapcount is only loosely connected to VMAs. It really counts the number of PUD/PTEs that point at parts of the memory.

Careful. Currently, you need to distinguish between total_mapcount(), page_trans_huge_mapcount() and page_mapcount(). Take a look at __page_mapcount() to be sure you really know what the mapcount "really" counts today ...

Yes, and the documentation above page_trans_huge_mapcount() tries to bring some clarity. Tries :)

Quoted 11 lines

(also I'm going to assume that when you said PUD you really mean PMD throughout)

Quoted 7 lines

If, under the PTL, you observe a mapcount of 1 then you know that the PUD/PTE you have under lock is the ONLY PUD/PTE that refers to this page and will remain so while the lock is held.

So, today the above ends up with a mapcount of 1 and when we take a COW fault we can re-use the page.

Quoted 8 lines
Quoted 3 lines

If the above ends up with a mapcount of 2 then COW will copy not re-use, which will cause unexpected data corruption in all those annoying side cases.

As I understood David's presentation yesterday, we actually have data corruption issues in all the annoying side cases with THPs in current upstream, so that's no worse than we have now. But let's see if we can avoid them.

Right, because the refcount is even more shaky ...

Quoted 11 lines

It feels like what we want from a COW perspective is a count of the number of MMs mapping a page, not the number of VMAs, PTEs or PMDs mapping the page. Right?

So here's a corner case ...

p = mmap(x, 2MB, PROT_READ|PROT_WRITE, ...): THP allocated mremap(p + 128K, 128K, 128K, MREMAP_MAYMOVE | MREMAP_FIXED, p + 2MB):
PMD split

(busy preparing and testing related patches, so I only skimmed over the discussion)

Whenever we have to go through an internal munmap (mmap, munmap, mremap), we would split the PMD and map the remainder using PTE. We place the huge page on the deferred split queue, where the actual compound page will get split ("THP split").

In move_page_tables() we perform the split_huge_pmd() as well, which would trigger in your example I think.

For anon pages, IIRC, there is no way to get more than one mapping per process for a single base page. "sharing" as in "shared anonymous pages" only applies between processes, not VMAs.

One anon base page can only be mapped once into a process ever. An anon base page can be mapped shared into multiple processes.

"The function returns the highest mapcount any one of the subpages has. If the return value is one, even if different processes are mapping different subpages of the transparent hugepage, they can all reuse it, because each process is reusing a different subpage."

So if you see "at least one subpage is mapped by more than one" and the page is anon shared, you have to split the PMD and trigger unsharing for exactly that subpage.

But it is indeed confusing ...

Quoted 1 line

Should mapcount be 1 or 2 at this point? Does the answer change if it's

The PMD was split. Each subpage is mapped exactly once.

page_trans_huge_mapcount() is supposed to return 1 because there is no sharing.

(Famous last words)

Signature · 3 lines
Thanks,

David / dhildenb
#6 lore raw
In reply to Matthew Wilcox
Quoted 8 lines · Matthew Wilcox
Quoted 7 lines · Jason Gunthorpe
Quoted 5 lines · Matthew Wilcox

p = mmap(x, 2MB, PROT_READ|PROT_WRITE, ...): THP allocated mprotect(p, 4KB, PROT_READ): THP split.

And in that case, I would say the THP now has mapcount of 2 because there are 2 VMAs mapping it.

Quoted 7 lines
Quoted 2 lines

At least today mapcount is only loosely connected to VMAs. It really counts the number of PUD/PTEs that point at parts of the memory.

Careful. Currently, you need to distinguish between total_mapcount(), page_trans_huge_mapcount() and page_mapcount(). Take a look at __page_mapcount() to be sure you really know what the mapcount "really" counts today ...

Right, I was mostly trying to describe one of the difficult problems all this different stuff is trying to solve.

Quoted 8 lines
Quoted 3 lines

If the above ends up with a mapcount of 2 then COW will copy not re-use, which will cause unexpected data corruption in all those annoying side cases.

As I understood David's presentation yesterday, we actually have data corruption issues in all the annoying side cases with THPs in current upstream, so that's no worse than we have now. But let's see if we can avoid them.

Possibly, I'm not sure :)

Quoted 3 lines

It feels like what we want from a COW perspective is a count of the number of MMs mapping a page, not the number of VMAs, PTEs or PMDs mapping the page. Right?

Interesting..

For the COW the interesting question is if wp_page_reuse() happens in do_wp_page(), and it looks like mapcount is only used to make that decision for anonymous pages. So, at least for COW's use of mapcount we can focus entirely on anon pages?

For anon pages.. At least the number of VMA's pointing to anon memory is a limit on map_count - I assume there is some way we can copy and double-map anonymous VMAs into the same mm? Still, if we can know the number of VMAs is 1 then we are safe to allow wp_page_reuse()

However, it needs to be more exact than that, if num VMAs > 1 we then have to query on a per-page granularity

Though, it seems interesting, if we knew how many anonymous VMA's pointed at an anonymous page (at a 4k granularity) that would replace mapcount for COW?

I wonder if we could somehow know that only 1 VMA is pointing at the pages as the normal fast path and if COW encounters more than 1 VMA it does some more expensive calculation?

Quoted 5 lines

p = mmap(x, 2MB, PROT_READ|PROT_WRITE, ...): THP allocated mremap(p + 128K, 128K, 128K, MREMAP_MAYMOVE | MREMAP_FIXED, p + 2MB):
PMD split

Should mapcount be 1 or 2 at this point?

If I read this right it should be 1 because each 4k page is pointed to by only 1 PTE/PMD. mremap moves, not copies..

Jason

#7 lore raw
In reply to Matthew Wilcox
Quoted 15 lines · Matthew Wilcox
Quoted 7 lines

Okay, so you will have mapcount == 2 or 3 for mprotect case above, not
512. But it doesn't help with answering question if the page can be re-used. You would need to do rmap walk to get the answer.

Note also that VMA lifecycle is different from page lifecycle:
MADV_DONTNEED removes mapping, but leaves VMA intact. Who would decrement mapcount here?

I think page_remove_rmap() does when called from zap_huge_pmd(). The tricky part is handling partial DONTNEED calls. eg we could have a 2MB page (if we're talking about shmem, it could even be mapped askew), MADV_DONTNEED the first 512KB, then MADV_DONTNEED the last 512KB, then finally MADV_DONTNEED the middle 1MB and only at the third call should the mapcount be decremented.

Quoted 4 lines

So a zap call has to check all the PTE/PMD entries in the range of the (folio intersect vma) to be sure that there are no more references to a part of this folio. It might not be terribly fast, but it probably won't be noticable compared to all the other costs of doing a munmap().

It's not only the area around unmapped. The rest of the page can be anywhere in the address space, thanks to mremap(2).

I think it would require full rmap, but I'm not sure. Or some very special tricks on mremap() or vma split. Looks like suffling complexity around, not reducing it.

Signature · 1 line
 Kirill A. Shutemov