Skip to contentSkip to conversation

kernel/fork: only call untrack_pfn_clear() on VMAs duplicated for fork()

PATCH Applied David Hildenbrand 2 participants 12 messages Apr 22, 2025 – Apr 24, 2025

Subsystems (13)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 kernel/fork: only call untrack_pfn_clear() on VMAs duplicated for fork()

Not intuitive, but vm_area_dup() located in kernel/fork.c is not only used for duplicating VMAs during fork(), but also for duplicating VMAs when splitting VMAs or when mremap()'ing them.

VM_PFNMAP mappings can at least get ordinarily mremap()'ed (no change in size) and apparently also shrunk during mremap(), which implies duplicating the VMA in __split_vma() first.

In case of ordinary mremap() (no change in size), we first duplicate the VMA in copy_vma_and_data()->copy_vma() to then call untrack_pfn_clear() on the old VMA: we effectively move the VM_PAT reservation. So the untrack_pfn_clear() call on the new VMA duplicating is wrong in that context.

Splitting of VMAs seems problematic, because we don't duplicate/adjust the reservation when splitting the VMA. Instead, in memtype_erase() -- called during zapping/munmap -- we shrink a reservation in case only the end address matches: Assume we split a VMA into A and B, both would share a reservation until B is unmapped.

So when unmapping B, the reservation would be updated to cover only A. When unmapping A, we would properly remove the now-shrunk reservation. That scenario describes the mremap() shrinking (old_size > new_size), where we split + unmap B, and the untrack_pfn_clear() on the new VMA when is wrong.

What if we manage to split a VM_PFNMAP VMA into A and B and unmap A first? It would be broken because we would never free the reservation. Likely, there are ways to trigger such a VMA split outside of mremap().

Affecting other VMA duplication was not intended, vm_area_dup() being used outside of kernel/fork.c was an oversight. So let's fix that for; how to handle VMA splits better should be investigated separately.

This was found by code inspection only, while staring at yet another VM_PAT problem.

Fixes dc84bc2aba85 ("x86/mm/pat: Fix VM_PAT handling when fork() fails in copy_page_range()")
Cc Andrew Morton <[email protected]>
Cc Lorenzo Stoakes <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Rik van Riel <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Linus Torvalds <[email protected]>

---

This VM_PAT code really wants me to scream at my computer. So far it didn't succeed, but I am close. Well, at least now I understand how it interacts with VMA splitting ...

---
 kernel/fork.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
kernel/fork.c +5 -4
index c4b26cd8998b8..168681fc4b25a 100644
@@ -498,10 +498,6 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig)
 	vma_numab_state_init(new);
 	dup_anon_vma_name(orig, new);
 
-	/* track_pfn_copy() will later take care of copying internal state. */
-	if (unlikely(new->vm_flags & VM_PFNMAP))
-		untrack_pfn_clear(new);
-
 	return new;
 }
 
@@ -672,6 +668,11 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
 		tmp = vm_area_dup(mpnt);
 		if (!tmp)
 			goto fail_nomem;
+
+		/* track_pfn_copy() will later take care of copying internal state. */
+		if (unlikely(tmp->vm_flags & VM_PFNMAP))
+			untrack_pfn_clear(tmp);
+
 		retval = vma_dup_policy(mpnt, tmp);
 		if (retval)
 			goto fail_nomem_policy;
Signature · 1 line
2.49.0
#1 lore raw
Quoted 26 lines · David Hildenbrand

Not intuitive, but vm_area_dup() located in kernel/fork.c is not only used for duplicating VMAs during fork(), but also for duplicating VMAs when splitting VMAs or when mremap()'ing them.

VM_PFNMAP mappings can at least get ordinarily mremap()'ed (no change in size) and apparently also shrunk during mremap(), which implies duplicating the VMA in __split_vma() first.

In case of ordinary mremap() (no change in size), we first duplicate the VMA in copy_vma_and_data()->copy_vma() to then call untrack_pfn_clear() on the old VMA: we effectively move the VM_PAT reservation. So the untrack_pfn_clear() call on the new VMA duplicating is wrong in that context.

Splitting of VMAs seems problematic, because we don't duplicate/adjust the reservation when splitting the VMA. Instead, in memtype_erase() -- called during zapping/munmap -- we shrink a reservation in case only the end address matches: Assume we split a VMA into A and B, both would share a reservation until B is unmapped.

So when unmapping B, the reservation would be updated to cover only A. When unmapping A, we would properly remove the now-shrunk reservation. That scenario describes the mremap() shrinking (old_size > new_size), where we split + unmap B, and the untrack_pfn_clear() on the new VMA when is wrong.

Quoted 3 lines

What if we manage to split a VM_PFNMAP VMA into A and B and unmap A first? It would be broken because we would never free the reservation. Likely, there are ways to trigger such a VMA split outside of mremap().

As expected ... with a simple reproducer that uses mprotect() to split such a VMA I can trigger

x86/PAT: pat_mremap:26448 freeing invalid memtype [mem 0x00000000-0x00000fff]

Signature · 3 lines
Cheers,

David / dhildenb
#2 lore raw
Quoted 31 lines · David Hildenbrand
Quoted 26 lines · David Hildenbrand

Not intuitive, but vm_area_dup() located in kernel/fork.c is not only used for duplicating VMAs during fork(), but also for duplicating VMAs when splitting VMAs or when mremap()'ing them.

VM_PFNMAP mappings can at least get ordinarily mremap()'ed (no change in size) and apparently also shrunk during mremap(), which implies duplicating the VMA in __split_vma() first.

In case of ordinary mremap() (no change in size), we first duplicate the VMA in copy_vma_and_data()->copy_vma() to then call untrack_pfn_clear() on the old VMA: we effectively move the VM_PAT reservation. So the untrack_pfn_clear() call on the new VMA duplicating is wrong in that context.

Splitting of VMAs seems problematic, because we don't duplicate/adjust the reservation when splitting the VMA. Instead, in memtype_erase() -- called during zapping/munmap -- we shrink a reservation in case only the end address matches: Assume we split a VMA into A and B, both would share a reservation until B is unmapped.

So when unmapping B, the reservation would be updated to cover only A. When unmapping A, we would properly remove the now-shrunk reservation. That scenario describes the mremap() shrinking (old_size > new_size), where we split + unmap B, and the untrack_pfn_clear() on the new VMA when is wrong.

Quoted 3 lines

What if we manage to split a VM_PFNMAP VMA into A and B and unmap A first? It would be broken because we would never free the reservation. Likely, there are ways to trigger such a VMA split outside of mremap().

Quoted 5 lines

As expected ... with a simple reproducer that uses mprotect() to split such a VMA I can trigger

x86/PAT: pat_mremap:26448 freeing invalid memtype [mem 0x00000000-0x00000fff]

Wow.

Might be worth adding a self test for this if not too difficult, even if as a skipped one w/comment just so we have it ready to go?

Quoted 7 lines

--
Cheers,

David / dhildenb

#3 lore raw
Quoted 35 lines · Lorenzo Stoakes
Quoted 34 lines · David Hildenbrand
Quoted 26 lines · David Hildenbrand

Not intuitive, but vm_area_dup() located in kernel/fork.c is not only used for duplicating VMAs during fork(), but also for duplicating VMAs when splitting VMAs or when mremap()'ing them.

VM_PFNMAP mappings can at least get ordinarily mremap()'ed (no change in size) and apparently also shrunk during mremap(), which implies duplicating the VMA in __split_vma() first.

In case of ordinary mremap() (no change in size), we first duplicate the VMA in copy_vma_and_data()->copy_vma() to then call untrack_pfn_clear() on the old VMA: we effectively move the VM_PAT reservation. So the untrack_pfn_clear() call on the new VMA duplicating is wrong in that context.

Splitting of VMAs seems problematic, because we don't duplicate/adjust the reservation when splitting the VMA. Instead, in memtype_erase() -- called during zapping/munmap -- we shrink a reservation in case only the end address matches: Assume we split a VMA into A and B, both would share a reservation until B is unmapped.

So when unmapping B, the reservation would be updated to cover only A. When unmapping A, we would properly remove the now-shrunk reservation. That scenario describes the mremap() shrinking (old_size > new_size), where we split + unmap B, and the untrack_pfn_clear() on the new VMA when is wrong.

Quoted 3 lines

What if we manage to split a VM_PFNMAP VMA into A and B and unmap A first? It would be broken because we would never free the reservation. Likely, there are ways to trigger such a VMA split outside of mremap().

As expected ... with a simple reproducer that uses mprotect() to split such a VMA I can trigger

Quoted 7 lines
Quoted 2 lines

x86/PAT: pat_mremap:26448 freeing invalid memtype [mem 0x00000000-0x00000fff]

Wow.

Might be worth adding a self test for this if not too difficult, even if as a skipped one w/comment just so we have it ready to go?

Yeah, I think it's as stupid as mmaping two pages worth of /dev/mem, and then munmapping only a single one. It's so easy to trigger that it makes me absolutely angry.

Signature · 3 lines
Cheers,

David / dhildenb
Reviewed-by on v1 1/1
#4 lore raw
In reply to David Hildenbrand

+cc Liam for the vma things, and because he adores PAT stuff ;)

Quoted 3 lines · David Hildenbrand

Not intuitive, but vm_area_dup() located in kernel/fork.c is not only used for duplicating VMAs during fork(), but also for duplicating VMAs when splitting VMAs or when mremap()'ing them.

Ugh this sucks, I really want to move a bunch of this stuff out of the fork code. we have some really nasty overlap there.

This definitely needs to be separate out. Perhaps I can take a look at that...

Quoted 4 lines

VM_PFNMAP mappings can at least get ordinarily mremap()'ed (no change in size) and apparently also shrunk during mremap(), which implies duplicating the VMA in __split_vma() first.

Yes, it appears we only disallow VM_PFNMAP on a remap if it is MREMAP_FIXED (implies MREMAP_MAYMOVE) to a new specific address _and_ we _increase_ the size of the VMA.

(as determined by vrm_implies_new_addr(), with resize_is_valid() explicitly disallowing MREMAP_DONTUNMAP).

Makes sense as VM_PFNMAP implies we map non-vm_normal_folio() stuff, which can't be faulted in, and thus we can't have unfaulted backing for it, but we can shrink safely.

Quoted 7 lines

In case of ordinary mremap() (no change in size), we first duplicate the VMA in copy_vma_and_data()->copy_vma() to then call untrack_pfn_clear() on the old VMA: we effectively move the VM_PAT reservation. So the untrack_pfn_clear() call on the new VMA duplicating is wrong in that context.

OK so we do:

copy_vma_and_data()
-> copy_vma()
-> vm_area_dup()
-> untrack_pfn_clear(new vma)

And:

copy_vma_and_data()
-> untrack_pfn_clear(old vma)

So we end up with... neither tracked. Fun.

Agreed this is incorrect.

Quoted 5 lines

Splitting of VMAs seems problematic, because we don't duplicate/adjust the reservation when splitting the VMA. Instead, in memtype_erase() -- called during zapping/munmap -- we shrink a reservation in case only the end address matches: Assume we split a VMA into A and B, both would share a reservation until B is unmapped.

Glorious. I really oppose us making radical changes to splitting logic to suit this one x86-specific feature.

Seems to me the change should be within PAT...

Quoted 10 lines

So when unmapping B, the reservation would be updated to cover only A. When unmapping A, we would properly remove the now-shrunk reservation. That scenario describes the mremap() shrinking (old_size > new_size), where we split + unmap B, and the untrack_pfn_clear() on the new VMA when is wrong.

What if we manage to split a VM_PFNMAP VMA into A and B and unmap A first? It would be broken because we would never free the reservation. Likely, there are ways to trigger such a VMA split outside of mremap().

This must have been a problem that already existed, and this is really quite plausible in reality, which makes me wonder, again, how much PAT is actually used in the wild.

I may be mistaken of course (happy to be corrected) and it's used very heavily and somehow this scenario doesn't occur.

We should definitely add a test for this anyway :) even if 'skip' for now while we figure out how to fix it (_without egregiously impacting split code_).

This can't happen in mremap() in any case as we are unmapping which happens to split then always remove the latter VMA.

OK so to get back into the split logic, this is:

shrink_vma() (in mm/mremap.c)
-> do_vmi_munmap()
-> do_vmi_align_munmap()
-> vms_gather_munmap_vmas()
-> __split_vma()
(later called from do_vmi_align_munmap())
-> vms_complete_munmap_vmas()
-> vms_clear_ptes()
-> unmap_vmas()
-> unmap_single_vma()
-> untrack_pfn()
-> free_pfn_range()
-> memtype_free()
-> memtype_erase()

As simple as that! ;)

Makes sense.

Quoted 4 lines

Affecting other VMA duplication was not intended, vm_area_dup() being used outside of kernel/fork.c was an oversight. So let's fix that for; how to handle VMA splits better should be investigated separately.

To reiterate, I think this should be handled within PAT itself, rather than result in changes to VMA code, unless it results in us adding sensible hooks there.

Quoted 3 lines

This was found by code inspection only, while staring at yet another VM_PAT problem.

My sympathies...

Quoted 14 lines
Fixes dc84bc2aba85 ("x86/mm/pat: Fix VM_PAT handling when fork() fails in copy_page_range()")
Cc Andrew Morton <[email protected]>
Cc Lorenzo Stoakes <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Rik van Riel <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Linus Torvalds <[email protected]>

Anyway, having gone through your excellent descriptions (albeit, feeling the same pain as you did :P), and looking at the logic, I agree this patch is correct, so:

Quoted 5 lines

---

This VM_PAT code really wants me to scream at my computer. So far it didn't succeed, but I am close. Well, at least now I understand how it interacts with VMA splitting ...

Well, I'm also quite scared of it, and indeed also relate ;) How heavily used is PAT? We do seem to constantly run into problems with getting it to behave itself wrt VMA code.

We recently had to remove some quite egregious hooks in VMA code which was a pain, is there a better way of doing this?

I really do hate this 'randomly call a function in various spots and do something specific for feature X' pattern that we use for hugetlb, uffd, this, and other stuff.

We need to use (carefully constrained!) generic hooks for this kind of thing.

At least tell me the mremap() refactor I did made this less horrible? ;)

Quoted 35 lines
---
 kernel/fork.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
kernel/fork.c +5 -5
index c4b26cd8998b8..168681fc4b25a 100644
@@ -498,10 +498,6 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig)
 	vma_numab_state_init(new);
 	dup_anon_vma_name(orig, new);
 
-	/* track_pfn_copy() will later take care of copying internal state. */
-	if (unlikely(new->vm_flags & VM_PFNMAP))
-		untrack_pfn_clear(new);
-
 	return new;
 }
 
@@ -672,6 +668,11 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
 		tmp = vm_area_dup(mpnt);
 		if (!tmp)
 			goto fail_nomem;
+
+		/* track_pfn_copy() will later take care of copying internal state. */
+		if (unlikely(tmp->vm_flags & VM_PFNMAP))
+			untrack_pfn_clear(tmp);
+
 		retval = vma_dup_policy(mpnt, tmp);
 		if (retval)
 			goto fail_nomem_policy;
--

2.49.0

#5 lore raw
Quoted 7 lines · Lorenzo Stoakes

+cc Liam for the vma things, and because he adores PAT stuff ;)

Quoted 3 lines · David Hildenbrand

Not intuitive, but vm_area_dup() located in kernel/fork.c is not only used for duplicating VMAs during fork(), but also for duplicating VMAs when splitting VMAs or when mremap()'ing them.

Quoted 5 lines

Ugh this sucks, I really want to move a bunch of this stuff out of the fork code. we have some really nasty overlap there.

This definitely needs to be separate out. Perhaps I can take a look at that...

Yes, it should likely live in ... vma.c ? :)

Quoted 47 lines
Quoted 5 lines

VM_PFNMAP mappings can at least get ordinarily mremap()'ed (no change in size) and apparently also shrunk during mremap(), which implies duplicating the VMA in __split_vma() first.

Yes, it appears we only disallow VM_PFNMAP on a remap if it is MREMAP_FIXED (implies MREMAP_MAYMOVE) to a new specific address _and_ we _increase_ the size of the VMA.

(as determined by vrm_implies_new_addr(), with resize_is_valid() explicitly disallowing MREMAP_DONTUNMAP).

Makes sense as VM_PFNMAP implies we map non-vm_normal_folio() stuff, which can't be faulted in, and thus we can't have unfaulted backing for it, but we can shrink safely.

Quoted 7 lines

In case of ordinary mremap() (no change in size), we first duplicate the VMA in copy_vma_and_data()->copy_vma() to then call untrack_pfn_clear() on the old VMA: we effectively move the VM_PAT reservation. So the untrack_pfn_clear() call on the new VMA duplicating is wrong in that context.

OK so we do:

copy_vma_and_data()
-> copy_vma()
-> vm_area_dup()
-> untrack_pfn_clear(new vma)

And:

copy_vma_and_data()
-> untrack_pfn_clear(old vma)

So we end up with... neither tracked. Fun.

Agreed this is incorrect.

Quoted 5 lines

Splitting of VMAs seems problematic, because we don't duplicate/adjust the reservation when splitting the VMA. Instead, in memtype_erase() -- called during zapping/munmap -- we shrink a reservation in case only the end address matches: Assume we split a VMA into A and B, both would share a reservation until B is unmapped.

Quoted 4 lines

Glorious. I really oppose us making radical changes to splitting logic to suit this one x86-specific feature.

Seems to me the change should be within PAT...

Yeah ...

Quoted 8 lines
Quoted 8 lines

So when unmapping B, the reservation would be updated to cover only A. When unmapping A, we would properly remove the now-shrunk reservation. That scenario describes the mremap() shrinking (old_size > new_size), where we split + unmap B, and the untrack_pfn_clear() on the new VMA when is wrong.

Quoted 7 lines
Quoted 3 lines

What if we manage to split a VM_PFNMAP VMA into A and B and unmap A first? It would be broken because we would never free the reservation. Likely, there are ways to trigger such a VMA split outside of mremap().

This must have been a problem that already existed, and this is really quite plausible in reality, which makes me wonder, again, how much PAT is actually used in the wild.

Probably people use it all the time, but we only fix stuff that we hit in practice.

Fun. :)

Quoted 28 lines

I may be mistaken of course (happy to be corrected) and it's used very heavily and somehow this scenario doesn't occur.

We should definitely add a test for this anyway :) even if 'skip' for now while we figure out how to fix it (_without egregiously impacting split code_).

This can't happen in mremap() in any case as we are unmapping which happens to split then always remove the latter VMA.

OK so to get back into the split logic, this is:

shrink_vma() (in mm/mremap.c)
-> do_vmi_munmap()
-> do_vmi_align_munmap()
-> vms_gather_munmap_vmas()
-> __split_vma()
(later called from do_vmi_align_munmap())
-> vms_complete_munmap_vmas()
-> vms_clear_ptes()
-> unmap_vmas()
-> unmap_single_vma()
-> untrack_pfn()
-> free_pfn_range()
-> memtype_free()
-> memtype_erase()

Quoted 1 line

As simple as that! ;)

My head started spinning when I followed that callchain :D

Quoted 11 lines

Makes sense.

Quoted 4 lines

Affecting other VMA duplication was not intended, vm_area_dup() being used outside of kernel/fork.c was an oversight. So let's fix that for; how to handle VMA splits better should be investigated separately.

To reiterate, I think this should be handled within PAT itself, rather than result in changes to VMA code, unless it results in us adding sensible hooks there.

Yes. I wish we could remove it; I'm afraid we can;t

[...]

Quoted 5 lines

Anyway, having gone through your excellent descriptions (albeit, feeling the same pain as you did :P), and looking at the logic, I agree this patch is correct, so:

Thanks!

Quoted 11 lines
Quoted 6 lines

---

This VM_PAT code really wants me to scream at my computer. So far it didn't succeed, but I am close. Well, at least now I understand how it interacts with VMA splitting ...

Well, I'm also quite scared of it, and indeed also relate ;) How heavily used is PAT? We do seem to constantly run into problems with getting it to behave itself wrt VMA code.

Quoted 6 lines

We recently had to remove some quite egregious hooks in VMA code which was a pain, is there a better way of doing this?

I really do hate this 'randomly call a function in various spots and do something specific for feature X' pattern that we use for hugetlb, uffd, this, and other stuff.

I hate it.

Probably the right way of attaching such metadata to a VMA would be remembering it alongside the VMA in a very simple way.

For example, when we perform a reservation we would allocate a refcounted object and assign it to the VMA (pointer, xarray, whatever).

Duplicating the VMA would increase the refcount. Freeing a VMA would decrease the refcount.

Once the refcount goes to zero, we undo the reservation and free the object.

We would not adjust a reservation on partial VMA unmap (split + unmap A or B), but I strongly assume that would just be fine as long as we undo the reservation once the refcount goes to 0.

There is one complication: this nasty memremap.c that abuses it for !vma handling. But that could be split out and handled differently.

Quoted 5 lines

We need to use (carefully constrained!) generic hooks for this kind of thing.

At least tell me the mremap() refactor I did made this less horrible? ;)

haha, absolutely :P

Signature · 3 lines
Cheers,

David / dhildenb
#6 lore raw
Quoted 8 lines · David Hildenbrand
Quoted 7 lines · Lorenzo Stoakes

+cc Liam for the vma things, and because he adores PAT stuff ;)

Quoted 3 lines · David Hildenbrand

Not intuitive, but vm_area_dup() located in kernel/fork.c is not only used for duplicating VMAs during fork(), but also for duplicating VMAs when splitting VMAs or when mremap()'ing them.

Quoted 7 lines
Quoted 5 lines

Ugh this sucks, I really want to move a bunch of this stuff out of the fork code. we have some really nasty overlap there.

This definitely needs to be separate out. Perhaps I can take a look at that...

Yes, it should likely live in ... vma.c ? :)

Yes indeed this would make most sense except if fork needs it vma.c/vma.h is (very intentionally) mm-internal so sadly can't live there in this case :(

Anyway on my todo will find a place for it in mm if possible... :)

Quoted 116 lines
Quoted 48 lines
Quoted 6 lines

VM_PFNMAP mappings can at least get ordinarily mremap()'ed (no change in size) and apparently also shrunk during mremap(), which implies duplicating the VMA in __split_vma() first.

Yes, it appears we only disallow VM_PFNMAP on a remap if it is MREMAP_FIXED (implies MREMAP_MAYMOVE) to a new specific address _and_ we _increase_ the size of the VMA.

(as determined by vrm_implies_new_addr(), with resize_is_valid() explicitly disallowing MREMAP_DONTUNMAP).

Makes sense as VM_PFNMAP implies we map non-vm_normal_folio() stuff, which can't be faulted in, and thus we can't have unfaulted backing for it, but we can shrink safely.

Quoted 7 lines

In case of ordinary mremap() (no change in size), we first duplicate the VMA in copy_vma_and_data()->copy_vma() to then call untrack_pfn_clear() on the old VMA: we effectively move the VM_PAT reservation. So the untrack_pfn_clear() call on the new VMA duplicating is wrong in that context.

OK so we do:

copy_vma_and_data()
-> copy_vma()
-> vm_area_dup()
-> untrack_pfn_clear(new vma)

And:

copy_vma_and_data()
-> untrack_pfn_clear(old vma)

So we end up with... neither tracked. Fun.

Agreed this is incorrect.

Quoted 5 lines

Splitting of VMAs seems problematic, because we don't duplicate/adjust the reservation when splitting the VMA. Instead, in memtype_erase() -- called during zapping/munmap -- we shrink a reservation in case only the end address matches: Assume we split a VMA into A and B, both would share a reservation until B is unmapped.

Quoted 4 lines

Glorious. I really oppose us making radical changes to splitting logic to suit this one x86-specific feature.

Seems to me the change should be within PAT...

Yeah ...

Quoted 8 lines
Quoted 8 lines

So when unmapping B, the reservation would be updated to cover only A. When unmapping A, we would properly remove the now-shrunk reservation. That scenario describes the mremap() shrinking (old_size > new_size), where we split + unmap B, and the untrack_pfn_clear() on the new VMA when is wrong.

Quoted 7 lines
Quoted 3 lines

What if we manage to split a VM_PFNMAP VMA into A and B and unmap A first? It would be broken because we would never free the reservation. Likely, there are ways to trigger such a VMA split outside of mremap().

This must have been a problem that already existed, and this is really quite plausible in reality, which makes me wonder, again, how much PAT is actually used in the wild.

Probably people use it all the time, but we only fix stuff that we hit in practice.

Fun. :)

Quoted 28 lines

I may be mistaken of course (happy to be corrected) and it's used very heavily and somehow this scenario doesn't occur.

We should definitely add a test for this anyway :) even if 'skip' for now while we figure out how to fix it (_without egregiously impacting split code_).

This can't happen in mremap() in any case as we are unmapping which happens to split then always remove the latter VMA.

OK so to get back into the split logic, this is:

shrink_vma() (in mm/mremap.c)
-> do_vmi_munmap()
-> do_vmi_align_munmap()
-> vms_gather_munmap_vmas()
-> __split_vma()
(later called from do_vmi_align_munmap())
-> vms_complete_munmap_vmas()
-> vms_clear_ptes()
-> unmap_vmas()
-> unmap_single_vma()
-> untrack_pfn()
-> free_pfn_range()
-> memtype_free()
-> memtype_erase()

Quoted 1 line

As simple as that! ;)

My head started spinning when I followed that callchain :D

Quoted 8 lines

Makes sense.

Quoted 4 lines

Affecting other VMA duplication was not intended, vm_area_dup() being used outside of kernel/fork.c was an oversight. So let's fix that for; how to handle VMA splits better should be investigated separately.

Quoted 5 lines
Quoted 3 lines

To reiterate, I think this should be handled within PAT itself, rather than result in changes to VMA code, unless it results in us adding sensible hooks there.

Yes. I wish we could remove it; I'm afraid we can;t

Sigh one of these... I guess I was being hopeful!

Quoted 10 lines

[...]

Quoted 5 lines

Anyway, having gone through your excellent descriptions (albeit, feeling the same pain as you did :P), and looking at the logic, I agree this patch is correct, so:

Thanks!

No probs!

Quoted 27 lines
Quoted 12 lines
Quoted 7 lines

---

This VM_PAT code really wants me to scream at my computer. So far it didn't succeed, but I am close. Well, at least now I understand how it interacts with VMA splitting ...

Well, I'm also quite scared of it, and indeed also relate ;) How heavily used is PAT? We do seem to constantly run into problems with getting it to behave itself wrt VMA code.

Quoted 6 lines

We recently had to remove some quite egregious hooks in VMA code which was a pain, is there a better way of doing this?

I really do hate this 'randomly call a function in various spots and do something specific for feature X' pattern that we use for hugetlb, uffd, this, and other stuff.

I hate it.

Probably the right way of attaching such metadata to a VMA would be remembering it alongside the VMA in a very simple way.

For example, when we perform a reservation we would allocate a refcounted object and assign it to the VMA (pointer, xarray, whatever).

Quoted 8 lines

Duplicating the VMA would increase the refcount. Freeing a VMA would decrease the refcount.

Once the refcount goes to zero, we undo the reservation and free the object.

We would not adjust a reservation on partial VMA unmap (split + unmap A or B), but I strongly assume that would just be fine as long as we undo the reservation once the refcount goes to 0.

Yeah this is a really good idea actually, almost kinda what refcounts are for haha...

The problem is we talk about this idly here, but neither of us wants to actually write PAT code I'd say, so this may go nowhere. But maybe one of us will get so frustrated that we do this anyway but still...

Then again - actually, is this something you are planning to tackle?

It's curious it's not come up before, maybe not usual for this to happen with VM_PFNMAP PAT mappings.

Quoted 3 lines

There is one complication: this nasty memremap.c that abuses it for !vma handling. But that could be split out and handled differently.

I've not really looked into this but pre-yuck...

Quoted 9 lines
Quoted 6 lines

We need to use (carefully constrained!) generic hooks for this kind of thing.

At least tell me the mremap() refactor I did made this less horrible? ;)

haha, absolutely :P

Good :)

Quoted 7 lines

--
Cheers,

David / dhildenb

Cheers, Lorenzo

#7 lore raw
Quoted 15 lines
Quoted 6 lines

Probably the right way of attaching such metadata to a VMA would be remembering it alongside the VMA in a very simple way.

For example, when we perform a reservation we would allocate a refcounted object and assign it to the VMA (pointer, xarray, whatever).

Quoted 8 lines

Duplicating the VMA would increase the refcount. Freeing a VMA would decrease the refcount.

Once the refcount goes to zero, we undo the reservation and free the object.

We would not adjust a reservation on partial VMA unmap (split + unmap A or B), but I strongly assume that would just be fine as long as we undo the reservation once the refcount goes to 0.

Quoted 8 lines

Yeah this is a really good idea actually, almost kinda what refcounts are for haha...

The problem is we talk about this idly here, but neither of us wants to actually write PAT code I'd say, so this may go nowhere. But maybe one of us will get so frustrated that we do this anyway but still...

Then again - actually, is this something you are planning to tackle?

I hate this much with that much passion that I'll give it a try for a couple of hours, as it might fix the other issues we are seeing. So far it looks like it cleans up stuff *beautifully*. Even VM_PAT can go ... :)

... and I think we still have space in vm_area_struct without increasing it beyond 192 bytes.

Signature · 3 lines
Cheers,

David / dhildenb
#8 lore raw
Quoted 22 lines · David Hildenbrand
Quoted 22 lines
Quoted 6 lines

Probably the right way of attaching such metadata to a VMA would be remembering it alongside the VMA in a very simple way.

For example, when we perform a reservation we would allocate a refcounted object and assign it to the VMA (pointer, xarray, whatever).

Quoted 8 lines

Duplicating the VMA would increase the refcount. Freeing a VMA would decrease the refcount.

Once the refcount goes to zero, we undo the reservation and free the object.

We would not adjust a reservation on partial VMA unmap (split + unmap A or B), but I strongly assume that would just be fine as long as we undo the reservation once the refcount goes to 0.

Yeah this is a really good idea actually, almost kinda what refcounts are for haha...

The problem is we talk about this idly here, but neither of us wants to actually write PAT code I'd say, so this may go nowhere. But maybe one of us will get so frustrated that we do this anyway but still...

Quoted 5 lines
Quoted 1 line

Then again - actually, is this something you are planning to tackle?

I hate this much with that much passion that I'll give it a try for a couple of hours, as it might fix the other issues we are seeing. So far it looks like it cleans up stuff *beautifully*. Even VM_PAT can go ... :)

To quote a film, let the hate flow through you :P

I mean I am very familiar with this - vma merge, anon_vma, etc. there's a pattern... ;)

Quoted 3 lines

... and I think we still have space in vm_area_struct without increasing it beyond 192 bytes.

Hm, so you're thinking of a general field in the VMA? I thought this would belong to the PAT object somehow?

Though getting rid of VM_PAT would be fantastic...

I wonder if a _general_ VMA ref count would be a bit much just for this case.

But maybe I misunderstand your approach :) Happy to obviously look and if not like some crazy thing just for PAT (you can understand why I would not like this) will be supportive :>)

Quoted 6 lines

--
Cheers,

David / dhildenb

Cheers, Lorenzo

#9 lore raw
Quoted 6 lines
Quoted 3 lines

... and I think we still have space in vm_area_struct without increasing it beyond 192 bytes.

Hm, so you're thinking of a general field in the VMA? I thought this would belong to the PAT object somehow?

It's glued to a VMA. The only alternative to using a VMA field would be looking it up for a VMA, storing it in an xarray etc ... ends up complicating stuff when there is no need to right now.

Quoted 5 lines

Though getting rid of VM_PAT would be fantastic...

I wonder if a _general_ VMA ref count would be a bit much just for this case.

I don't think it would be helpful for this case. It's much more similar to the anon VMA name (that also has its own kref)

Quoted 4 lines

But maybe I misunderstand your approach :) Happy to obviously look and if not like some crazy thing just for PAT (you can understand why I would not like this) will be supportive :>)

This is something quick (well, longer than I wish it would take) that seems to work. There are smaller pat-internal cleanups to be had on top of this, and the new functions shall be documented.

Observe how:
* We remove VM_PAT and that weird VM flags manipulation + "locked" flag
* We remove any traces of the nasty tracking handling from mremap+fork code
* Just like anon_vma_name, it hooks into vm_area_dup()/vm_area_free().
* We remove the page table lookup via get_pat_info()->... completely
* We remove the VMA parameter from PAT code completely
* We reduce the track/untrack/sanitize interface to 3 functions

From 4cf8b2a2e60220c5b438adf920d75cba3af50ab4 Mon Sep 17 00:00:00 2001
From: David Hildenbrand <[email protected]>
Date: Thu, 24 Apr 2025 12:06:15 +0200
Subject: [PATCH] mm: rewrite pfnmap tracking

---
 arch/x86/mm/pat/memtype.c      | 155 ++-------------------------------
 drivers/gpu/drm/i915/i915_mm.c |   4 +-
 include/linux/mm.h             |   4 +-
 include/linux/mm_inline.h      |   2 +
 include/linux/mm_types.h       |  11 +++
 include/linux/pgtable.h        |  71 ++-------------
 include/trace/events/mmflags.h |   4 +-
 kernel/fork.c                  |  54 ++++++++++--
 mm/huge_memory.c               |   7 +-
 mm/io-mapping.c                |   2 +-
 mm/memory.c                    |  93 +++++++++++++++-----
 mm/memremap.c                  |   8 +-
 mm/mremap.c                    |   4 -
 13 files changed, 162 insertions(+), 257 deletions(-)
arch/x86/mm/pat/memtype.c +6 -149
index 72d8cbc611583..237c7e5e9d9aa 100644
@@ -932,124 +932,14 @@ static void free_pfn_range(u64 paddr, unsigned long size)
 		memtype_free(paddr, paddr + size);
 }
 
-static int follow_phys(struct vm_area_struct *vma, unsigned long *prot,
-		resource_size_t *phys)
-{
-	struct follow_pfnmap_args args = { .vma = vma, .address = vma->vm_start };
-
-	if (follow_pfnmap_start(&args))
-		return -EINVAL;
-
-	/* Never return PFNs of anon folios in COW mappings. */
-	if (!args.special) {
-		follow_pfnmap_end(&args);
-		return -EINVAL;
-	}
-
-	*prot = pgprot_val(args.pgprot);
-	*phys = (resource_size_t)args.pfn << PAGE_SHIFT;
-	follow_pfnmap_end(&args);
-	return 0;
-}
-
-static int get_pat_info(struct vm_area_struct *vma, resource_size_t *paddr,
-		pgprot_t *pgprot)
-{
-	unsigned long prot;
-
-	VM_WARN_ON_ONCE(!(vma->vm_flags & VM_PAT));
-
-	/*
-	 * We need the starting PFN and cachemode used for track_pfn_remap()
-	 * that covered the whole VMA. For most mappings, we can obtain that
-	 * information from the page tables. For COW mappings, we might now
-	 * suddenly have anon folios mapped and follow_phys() will fail.
-	 *
-	 * Fallback to using vma->vm_pgoff, see remap_pfn_range_notrack(), to
-	 * detect the PFN. If we need the cachemode as well, we're out of luck
-	 * for now and have to fail fork().
-	 */
-	if (!follow_phys(vma, &prot, paddr)) {
-		if (pgprot)
-			*pgprot = __pgprot(prot);
-		return 0;
-	}
-	if (is_cow_mapping(vma->vm_flags)) {
-		if (pgprot)
-			return -EINVAL;
-		*paddr = (resource_size_t)vma->vm_pgoff << PAGE_SHIFT;
-		return 0;
-	}
-	WARN_ON_ONCE(1);
-	return -EINVAL;
-}
-
-int track_pfn_copy(struct vm_area_struct *dst_vma,
-		struct vm_area_struct *src_vma, unsigned long *pfn)
-{
-	const unsigned long vma_size = src_vma->vm_end - src_vma->vm_start;
-	resource_size_t paddr;
-	pgprot_t pgprot;
-	int rc;
-
-	if (!(src_vma->vm_flags & VM_PAT))
-		return 0;
-
-	/*
-	 * Duplicate the PAT information for the dst VMA based on the src
-	 * VMA.
-	 */
-	if (get_pat_info(src_vma, &paddr, &pgprot))
-		return -EINVAL;
-	rc = reserve_pfn_range(paddr, vma_size, &pgprot, 1);
-	if (rc)
-		return rc;
-
-	/* Reservation for the destination VMA succeeded. */
-	vm_flags_set(dst_vma, VM_PAT);
-	*pfn = PHYS_PFN(paddr);
-	return 0;
-}
-
-void untrack_pfn_copy(struct vm_area_struct *dst_vma, unsigned long pfn)
-{
-	untrack_pfn(dst_vma, pfn, dst_vma->vm_end - dst_vma->vm_start, true);
-	/*
-	 * Reservation was freed, any copied page tables will get cleaned
-	 * up later, but without getting PAT involved again.
-	 */
-}
-
-/*
- * prot is passed in as a parameter for the new mapping. If the vma has
- * a linear pfn mapping for the entire range, or no vma is provided,
- * reserve the entire pfn + size range with single reserve_pfn_range
- * call.
- */
-int track_pfn_remap(struct vm_area_struct *vma, pgprot_t *prot,
-		    unsigned long pfn, unsigned long addr, unsigned long size)
+int pfnmap_sanitize(unsigned long pfn, unsigned long size, pgprot_t *prot)
 {
 	resource_size_t paddr = (resource_size_t)pfn << PAGE_SHIFT;
 	enum page_cache_mode pcm;
 
-	/* reserve the whole chunk starting from paddr */
-	if (!vma || (addr == vma->vm_start
-				&& size == (vma->vm_end - vma->vm_start))) {
-		int ret;
-
-		ret = reserve_pfn_range(paddr, size, prot, 0);
-		if (ret == 0 && vma)
-			vm_flags_set(vma, VM_PAT);
-		return ret;
-	}
-
 	if (!pat_enabled())
 		return 0;
 
-	/*
-	 * For anything smaller than the vma size we set prot based on the
-	 * lookup.
-	 */
 	pcm = lookup_memtype(paddr);
 
 	/* Check memtype for the remaining pages */
@@ -1066,51 +956,18 @@ int track_pfn_remap(struct vm_area_struct *vma, pgprot_t *prot,
 	return 0;
 }
 
-void track_pfn_insert(struct vm_area_struct *vma, pgprot_t *prot, pfn_t pfn)
+int pfnmap_track(unsigned long pfn, unsigned long size, pgprot_t *prot)
 {
-	enum page_cache_mode pcm;
-
-	if (!pat_enabled())
-		return;
+	const resource_size_t paddr = (resource_size_t)pfn << PAGE_SHIFT;
 
-	/* Set prot based on lookup */
-	pcm = lookup_memtype(pfn_t_to_phys(pfn));
-	*prot = __pgprot((pgprot_val(*prot) & (~_PAGE_CACHE_MASK)) |
-			 cachemode2protval(pcm));
+	return reserve_pfn_range(paddr, size, prot, 0);
 }
 
-/*
- * untrack_pfn is called while unmapping a pfnmap for a region.
- * untrack can be called for a specific region indicated by pfn and size or
- * can be for the entire vma (in which case pfn, size are zero).
- */
-void untrack_pfn(struct vm_area_struct *vma, unsigned long pfn,
-		 unsigned long size, bool mm_wr_locked)
+void pfnmap_untrack(unsigned long pfn, unsigned long size)
 {
-	resource_size_t paddr;
+	const resource_size_t paddr = (resource_size_t)pfn << PAGE_SHIFT;
 
-	if (vma && !(vma->vm_flags & VM_PAT))
-		return;
-
-	/* free the chunk starting from pfn or the whole chunk */
-	paddr = (resource_size_t)pfn << PAGE_SHIFT;
-	if (!paddr && !size) {
-		if (get_pat_info(vma, &paddr, NULL))
-			return;
-		size = vma->vm_end - vma->vm_start;
-	}
 	free_pfn_range(paddr, size);
-	if (vma) {
-		if (mm_wr_locked)
-			vm_flags_clear(vma, VM_PAT);
-		else
-			__vm_flags_mod(vma, 0, VM_PAT);
-	}
-}
-
-void untrack_pfn_clear(struct vm_area_struct *vma)
-{
-	vm_flags_clear(vma, VM_PAT);
 }
 
 pgprot_t pgprot_writecombine(pgprot_t prot)
drivers/gpu/drm/i915/i915_mm.c +2 -2
index 76e2801619f09..c33bd3d830699 100644
@@ -100,7 +100,7 @@ int remap_io_mapping(struct vm_area_struct *vma,
 
 	GEM_BUG_ON((vma->vm_flags & EXPECTED_FLAGS) != EXPECTED_FLAGS);
 
-	/* We rely on prevalidation of the io-mapping to skip track_pfn(). */
+	/* We rely on prevalidation of the io-mapping to skip pfnmap tracking. */
 	r.mm = vma->vm_mm;
 	r.pfn = pfn;
 	r.prot = __pgprot((pgprot_val(iomap->prot) & _PAGE_CACHE_MASK) |
@@ -140,7 +140,7 @@ int remap_io_sg(struct vm_area_struct *vma,
 	};
 	int err;
 
-	/* We rely on prevalidation of the io-mapping to skip track_pfn(). */
+	/* We rely on prevalidation of the io-mapping to skip pfnmap tracking. */
 	GEM_BUG_ON((vma->vm_flags & EXPECTED_FLAGS) != EXPECTED_FLAGS);
 
 	while (offset >= r.sgt.max >> PAGE_SHIFT) {
include/linux/mm.h +1 -3
index bf55206935c46..1dc7df6ff38e9 100644
@@ -356,9 +356,7 @@ extern unsigned int kobjsize(const void *objp);
 # define VM_SHADOW_STACK	VM_NONE
 #endif
 
-#if defined(CONFIG_X86)
-# define VM_PAT		VM_ARCH_1	/* PAT reserves whole VMA at once (x86) */
-#elif defined(CONFIG_PPC64)
+#if defined(CONFIG_PPC64)
 # define VM_SAO		VM_ARCH_1	/* Strong Access Ordering (powerpc) */
 #elif defined(CONFIG_PARISC)
 # define VM_GROWSUP	VM_ARCH_1
include/linux/mm_inline.h +2 -0
index f9157a0c42a5c..89b518ff097e6 100644
@@ -447,6 +447,8 @@ static inline bool anon_vma_name_eq(struct anon_vma_name *anon_name1,
 
 #endif  /* CONFIG_ANON_VMA_NAME */
 
+void pfnmap_track_ctx_release(struct kref *ref);
+
 static inline void init_tlb_flush_pending(struct mm_struct *mm)
 {
 	atomic_set(&mm->tlb_flush_pending, 0);
include/linux/mm_types.h +11 -0
index 56d07edd01f91..91124761cfda8 100644
@@ -764,6 +764,14 @@ struct vma_numab_state {
 	int prev_scan_seq;
 };
 
+#ifdef __HAVE_PFNMAP_TRACKING
+struct pfnmap_track_ctx {
+	struct kref kref;
+	unsigned long pfn;
+	unsigned long size;
+};
+#endif
+
 /*
  * This struct describes a virtual memory area. There is one of these
  * per VM-area/task. A VM area is any part of the process virtual memory
@@ -877,6 +885,9 @@ struct vm_area_struct {
 	struct anon_vma_name *anon_name;
 #endif
 	struct vm_userfaultfd_ctx vm_userfaultfd_ctx;
+#ifdef __HAVE_PFNMAP_TRACKING
+	struct pfnmap_track_ctx *pfnmap_track_ctx;
+#endif
 } __randomize_layout;
 
 #ifdef CONFIG_NUMA
include/linux/pgtable.h +7 -64

89 lines not on this page · raw message

include/trace/events/mmflags.h +1 -3
index 15aae955a10bf..aa441f593e9a6 100644
@@ -172,9 +172,7 @@ IF_HAVE_PG_ARCH_3(arch_3)
 	__def_pageflag_names						\
 	) : "none"
 
-#if defined(CONFIG_X86)
-#define __VM_ARCH_SPECIFIC_1 {VM_PAT,     "pat"           }
-#elif defined(CONFIG_PPC64)
+#if defined(CONFIG_PPC64)
 #define __VM_ARCH_SPECIFIC_1 {VM_SAO,     "sao"           }
 #elif defined(CONFIG_PARISC)
 #define __VM_ARCH_SPECIFIC_1 {VM_GROWSUP,	"growsup"	}
kernel/fork.c +49 -5

77 lines not on this page · raw message

mm/huge_memory.c +5 -2
index 2a47682d1ab77..9ad6a0a8f0089 100644
@@ -1456,7 +1456,9 @@ vm_fault_t vmf_insert_pfn_pmd(struct vm_fault *vmf, pfn_t pfn, bool write)
 			return VM_FAULT_OOM;
 	}
 
-	track_pfn_insert(vma, &pgprot, pfn);
+	/* TODO: we should check the whole range and handle errors. */
+	pfnmap_sanitize(pfn_t_to_pfn(pfn), PAGE_SIZE, &pgprot);
+
 	ptl = pmd_lock(vma->vm_mm, vmf->pmd);
 	error = insert_pfn_pmd(vma, addr, vmf->pmd, pfn, pgprot, write,
 			pgtable);
@@ -1578,7 +1580,8 @@ vm_fault_t vmf_insert_pfn_pud(struct vm_fault *vmf, pfn_t pfn, bool write)
 	if (addr < vma->vm_start || addr >= vma->vm_end)
 		return VM_FAULT_SIGBUS;
 
-	track_pfn_insert(vma, &pgprot, pfn);
+	/* TODO: we should check the whole range and handle errors. */
+	pfnmap_sanitize(pfn_t_to_pfn(pfn), PAGE_SIZE, &pgprot);
 
 	ptl = pud_lock(vma->vm_mm, vmf->pud);
 	insert_pfn_pud(vma, addr, vmf->pud, pfn, write);
mm/io-mapping.c +1 -1
index 01b3627999304..7266441ad0834 100644
@@ -21,7 +21,7 @@ int io_mapping_map_user(struct io_mapping *iomap, struct vm_area_struct *vma,
 	if (WARN_ON_ONCE((vma->vm_flags & expected_flags) != expected_flags))
 		return -EINVAL;
 
-	/* We rely on prevalidation of the io-mapping to skip track_pfn(). */
+	/* We rely on prevalidation of the io-mapping to skip pfnmap tracking. */
 	return remap_pfn_range_notrack(vma, addr, pfn, size,
 		__pgprot((pgprot_val(iomap->prot) & _PAGE_CACHE_MASK) |
 			 (pgprot_val(vma->vm_page_prot) & ~_PAGE_CACHE_MASK)));
mm/memory.c +73 -20

149 lines not on this page · raw message

mm/memremap.c +4 -4
index 2aebc1b192da9..c417c843e9b1f 100644
@@ -130,7 +130,7 @@ static void pageunmap_range(struct dev_pagemap *pgmap, int range_id)
 	}
 	mem_hotplug_done();
 
-	untrack_pfn(NULL, PHYS_PFN(range->start), range_len(range), true);
+	pfnmap_untrack(PHYS_PFN(range->start), range_len(range));
 	pgmap_array_delete(range);
 }
 
@@ -211,8 +211,8 @@ static int pagemap_range(struct dev_pagemap *pgmap, struct mhp_params *params,
 	if (nid < 0)
 		nid = numa_mem_id();
 
-	error = track_pfn_remap(NULL, &params->pgprot, PHYS_PFN(range->start), 0,
-			range_len(range));
+	error = pfnmap_track(PHYS_PFN(range->start), range_len(range),
+			     &params->pgprot);
 	if (error)
 		goto err_pfn_remap;
 
@@ -277,7 +277,7 @@ static int pagemap_range(struct dev_pagemap *pgmap, struct mhp_params *params,
 	if (!is_private)
 		kasan_remove_zero_shadow(__va(range->start), range_len(range));
 err_kasan:
-	untrack_pfn(NULL, PHYS_PFN(range->start), range_len(range), true);
+	pfnmap_untrack(PHYS_PFN(range->start), range_len(range));
 err_pfn_remap:
 	pgmap_array_delete(range);
 	return error;
mm/mremap.c +0 -4
index 7db9da609c84f..6e78e02f74bd3 100644
@@ -1191,10 +1191,6 @@ static int copy_vma_and_data(struct vma_remap_struct *vrm,
 	if (is_vm_hugetlb_page(vma))
 		clear_vma_resv_huge_pages(vma);
 
-	/* Tell pfnmap has moved from this vma */
-	if (unlikely(vma->vm_flags & VM_PFNMAP))
-		untrack_pfn_clear(vma);
-
 	*new_vma_ptr = new_vma;
 	return err;
 }
Signature · 7 lines
2.49.0


-- 
Cheers,

David / dhildenb
#10 lore raw
Quoted 10 lines · David Hildenbrand
Quoted 6 lines
Quoted 3 lines

... and I think we still have space in vm_area_struct without increasing it beyond 192 bytes.

Hm, so you're thinking of a general field in the VMA? I thought this would belong to the PAT object somehow?

It's glued to a VMA. The only alternative to using a VMA field would be looking it up for a VMA, storing it in an xarray etc ... ends up complicating stuff when there is no need to right now.

Yeah no, that'd be crazy.

I guess we are stuck with this so need to think of a sensible way forward.

Quoted 9 lines
Quoted 6 lines

Though getting rid of VM_PAT would be fantastic...

I wonder if a _general_ VMA ref count would be a bit much just for this case.

I don't think it would be helpful for this case. It's much more similar to the anon VMA name (that also has its own kref)

Ahh I see, adding a field like this, that makes a lot more sense, obviously could put behind appropriate CONFIG_ flag.

Given cacheline alignment probably have a bit of wiggle room there to avoid increasing VMA size also.

Quoted 11 lines
Quoted 5 lines

But maybe I misunderstand your approach :) Happy to obviously look and if not like some crazy thing just for PAT (you can understand why I would not like this) will be supportive :>)

This is something quick (well, longer than I wish it would take) that seems to work. There are smaller pat-internal cleanups to be had on top of this, and the new functions shall be documented.

Quoted 7 lines

Observe how:
* We remove VM_PAT and that weird VM flags manipulation + "locked" flag
* We remove any traces of the nasty tracking handling from mremap+fork code
* Just like anon_vma_name, it hooks into vm_area_dup()/vm_area_free().
* We remove the page table lookup via get_pat_info()->... completely
* We remove the VMA parameter from PAT code completely
* We reduce the track/untrack/sanitize interface to 3 functions

Yeah this is all lovely!

OK this should hopefully be workable then!

#11 lore raw
Quoted 17 lines
Quoted 9 lines
Quoted 3 lines

But maybe I misunderstand your approach :) Happy to obviously look and if not like some crazy thing just for PAT (you can understand why I would not like this) will be supportive :>)

This is something quick (well, longer than I wish it would take) that seems to work. There are smaller pat-internal cleanups to be had on top of this, and the new functions shall be documented.

Quoted 7 lines

Observe how:
* We remove VM_PAT and that weird VM flags manipulation + "locked" flag
* We remove any traces of the nasty tracking handling from mremap+fork code
* Just like anon_vma_name, it hooks into vm_area_dup()/vm_area_free().
* We remove the page table lookup via get_pat_info()->... completely
* We remove the VMA parameter from PAT code completely
* We reduce the track/untrack/sanitize interface to 3 functions

Quoted 3 lines

Yeah this is all lovely!

OK this should hopefully be workable then!

Okay, let me polish that up (and see if there is any reasonable way to split it up), and write some doc+descriptions .. and do some more testing.

The VMA split reproducer is definitely happy with this already.

Signature · 3 lines
Cheers,

David / dhildenb