Skip to contentSkip to conversation

mm: do not increment pgfault stats when page fault handler retries

PATCH Superseded Suren Baghdasaryan 3 participants 8 messages Apr 14, 2023 – Apr 15, 2023

Subsystems (2)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Suren Baghdasaryan

author reviewer
#0 lore raw

PATCH 1/1 mm: do not increment pgfault stats when page fault handler retries

If the page fault handler requests a retry, we will count the fault multiple times. This is a relatively harmless problem as the retry paths are not often requested, and the only user-visible problem is that the fault counter will be slightly higher than it should be. Nevertheless, userspace only took one fault, and should not see the fact that the kernel had to retry the fault multiple times.

Fixes 6b4c9f446981 ("filemap: drop the mmap_sem for all blocking operations")

---
Patch applies cleanly over linux-next and mm-unstable

 mm/memory.c | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)
mm/memory.c +10 -6
index 1c5b231fe6e3..d88f370eacd1 100644
@@ -5212,17 +5212,16 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
 
 	__set_current_state(TASK_RUNNING);
 
-	count_vm_event(PGFAULT);
-	count_memcg_event_mm(vma->vm_mm, PGFAULT);
-
 	ret = sanitize_fault_flags(vma, &flags);
 	if (ret)
-		return ret;
+		goto out;
 
 	if (!arch_vma_access_permitted(vma, flags & FAULT_FLAG_WRITE,
 					    flags & FAULT_FLAG_INSTRUCTION,
-					    flags & FAULT_FLAG_REMOTE))
-		return VM_FAULT_SIGSEGV;
+					    flags & FAULT_FLAG_REMOTE)) {
+		ret = VM_FAULT_SIGSEGV;
+		goto out;
+	}
 
 	/*
 	 * Enable the memcg OOM handling for faults triggered in user
@@ -5253,6 +5252,11 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
 	}
 
 	mm_account_fault(regs, address, flags, ret);
+out:
+	if (!(ret & VM_FAULT_RETRY)) {
+		count_vm_event(PGFAULT);
+		count_memcg_event_mm(vma->vm_mm, PGFAULT);
+	}
 
 	return ret;
 }
Signature · 1 line
2.40.0.634.g4ca3ef3211-goog
#1 lore raw
Quoted 8 lines · Suren Baghdasaryan

If the page fault handler requests a retry, we will count the fault multiple times. This is a relatively harmless problem as the retry paths are not often requested, and the only user-visible problem is that the fault counter will be slightly higher than it should be. Nevertheless, userspace only took one fault, and should not see the fact that the kernel had to retry the fault multiple times.

Fixes 6b4c9f446981 ("filemap: drop the mmap_sem for all blocking operations")

I know I suggested this fixes line, but I think it's actually been here much longer, perhaps since

Fixes d065bd810b6d ("mm: retry page fault when blocking on disk transfer")

Michel, what do you think?

Quoted 50 lines

---
Patch applies cleanly over linux-next and mm-unstable

 mm/memory.c | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)
mm/memory.c +10 -6
index 1c5b231fe6e3..d88f370eacd1 100644
@@ -5212,17 +5212,16 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
 
 	__set_current_state(TASK_RUNNING);
 
-	count_vm_event(PGFAULT);
-	count_memcg_event_mm(vma->vm_mm, PGFAULT);
-
 	ret = sanitize_fault_flags(vma, &flags);
 	if (ret)
-		return ret;
+		goto out;
 
 	if (!arch_vma_access_permitted(vma, flags & FAULT_FLAG_WRITE,
 					    flags & FAULT_FLAG_INSTRUCTION,
-					    flags & FAULT_FLAG_REMOTE))
-		return VM_FAULT_SIGSEGV;
+					    flags & FAULT_FLAG_REMOTE)) {
+		ret = VM_FAULT_SIGSEGV;
+		goto out;
+	}
 
 	/*
 	 * Enable the memcg OOM handling for faults triggered in user
@@ -5253,6 +5252,11 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
 	}
 
 	mm_account_fault(regs, address, flags, ret);
+out:
+	if (!(ret & VM_FAULT_RETRY)) {
+		count_vm_event(PGFAULT);
+		count_memcg_event_mm(vma->vm_mm, PGFAULT);
+	}
 
 	return ret;
 }
Signature · 1 line
2.40.0.634.g4ca3ef3211-goog
#2 lore raw
In reply to Suren Baghdasaryan

Hi, Suren,

Quoted 44 lines · Suren Baghdasaryan

If the page fault handler requests a retry, we will count the fault multiple times. This is a relatively harmless problem as the retry paths are not often requested, and the only user-visible problem is that the fault counter will be slightly higher than it should be. Nevertheless, userspace only took one fault, and should not see the fact that the kernel had to retry the fault multiple times.

Fixes 6b4c9f446981 ("filemap: drop the mmap_sem for all blocking operations")

---
Patch applies cleanly over linux-next and mm-unstable

 mm/memory.c | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)
mm/memory.c +5 -6
index 1c5b231fe6e3..d88f370eacd1 100644
@@ -5212,17 +5212,16 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
 
 	__set_current_state(TASK_RUNNING);
 
-	count_vm_event(PGFAULT);
-	count_memcg_event_mm(vma->vm_mm, PGFAULT);
-
 	ret = sanitize_fault_flags(vma, &flags);
 	if (ret)
-		return ret;
+		goto out;
 
 	if (!arch_vma_access_permitted(vma, flags & FAULT_FLAG_WRITE,
 					    flags & FAULT_FLAG_INSTRUCTION,
-					    flags & FAULT_FLAG_REMOTE))
-		return VM_FAULT_SIGSEGV;
+					    flags & FAULT_FLAG_REMOTE)) {
+		ret = VM_FAULT_SIGSEGV;
+		goto out;
+	}
 
 	/*
 	 * Enable the memcg OOM handling for faults triggered in user

mm/memory.c vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, · line 5254

Quoted 3 lines
mm/memory.c +0 -0
@@ -5253,6 +5252,11 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
 	}
 
 	mm_account_fault(regs, address, flags, ret);

Here is the mm_account_fault() function taking care of some other accountings. Perhaps good to put things into it?

It also already ignores invalid faults:

	if (ret & (VM_FAULT_ERROR | VM_FAULT_RETRY))
		return;

I see that you may also want to account for sigbus, however I really don't know why. Explanations would be great when it would matter. So far it makes sense to me if we skip both RETRY or ERROR cases.

mm/memory.c vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, · line 5258

Quoted 4 lines
mm/memory.c +4 -0
+out:
+	if (!(ret & VM_FAULT_RETRY)) {
+		count_vm_event(PGFAULT);
+		count_memcg_event_mm(vma->vm_mm, PGFAULT);

There is one thing worth noticing is here vma may or may not be valid depending on the retval of the fault.

RETRY is exactly one of the cases that accessing vma may be unsafe due to releasing of mmap read lock. The other one is the recently added VM_FAULT_COMPLETE. So if we want to move this chunk (or any vma reference) to be later we need to consider a valid vma / mm being there first, or we're prone to accessing a vma that has already been released, I think.

Quoted 8 lines
mm/memory.c +1 -0
+	}
 
 	return ret;
 }
Signature · 1 line
2.40.0.634.g4ca3ef3211-goog

Thanks,

Signature · 1 line
Peter Xu
author reviewer
#3 lore raw
Quoted 2 lines · Peter Xu

Hi, Suren,

Hi Peter,

Quoted 48 lines · Suren Baghdasaryan
Quoted 46 lines

If the page fault handler requests a retry, we will count the fault multiple times. This is a relatively harmless problem as the retry paths are not often requested, and the only user-visible problem is that the fault counter will be slightly higher than it should be. Nevertheless, userspace only took one fault, and should not see the fact that the kernel had to retry the fault multiple times.

Fixes 6b4c9f446981 ("filemap: drop the mmap_sem for all blocking operations")

---
Patch applies cleanly over linux-next and mm-unstable

 mm/memory.c | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)
mm/memory.c +5 -6
index 1c5b231fe6e3..d88f370eacd1 100644
@@ -5212,17 +5212,16 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
 
      __set_current_state(TASK_RUNNING);
 
-     count_vm_event(PGFAULT);
-     count_memcg_event_mm(vma->vm_mm, PGFAULT);
-
      ret = sanitize_fault_flags(vma, &flags);
      if (ret)
-             return ret;
+             goto out;
 
      if (!arch_vma_access_permitted(vma, flags & FAULT_FLAG_WRITE,
                                          flags & FAULT_FLAG_INSTRUCTION,
-                                         flags & FAULT_FLAG_REMOTE))
-             return VM_FAULT_SIGSEGV;
+                                         flags & FAULT_FLAG_REMOTE)) {
+             ret = VM_FAULT_SIGSEGV;
+             goto out;
+     }
 
      /*
       * Enable the memcg OOM handling for faults triggered in user
@@ -5253,6 +5252,11 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
      }
 
Quoted 4 lines

mm/memory.c vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, · line 5254

Quoted 1 line
mm/memory.c +0 -0
      mm_account_fault(regs, address, flags, ret);

Here is the mm_account_fault() function taking care of some other accountings. Perhaps good to put things into it?

That seems appropriate. Let me take a closer look.

Quoted 5 lines

It also already ignores invalid faults:

if (ret & (VM_FAULT_ERROR | VM_FAULT_RETRY))
return;

Can there be a case of (!VM_FAULT_ERROR && VM_FAULT_RETRY) - basically we need to retry but no errors happened? If so then this condition would double-count pagefaults in such cases. If such return code is impossible then it's the same as checking for VM_FAULT_RETRY.

Quoted 4 lines

I see that you may also want to account for sigbus, however I really don't know why. Explanations would be great when it would matter. So far it makes sense to me if we skip both RETRY or ERROR cases.

Accounting in case of a sigbus is not affected by this patch I think. We account for sigbus or any other error cases because there was a pagefault and we need to account for it. Whether we failed to handle it or not should not affect the count. We skip the retry case because we know the same fault will be retried. If we don't skip then we will double-count this fault.

Quoted 6 lines

mm/memory.c vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, · line 5258

Quoted 5 lines
mm/memory.c +4 -0
+out:
+     if (!(ret & VM_FAULT_RETRY)) {
+             count_vm_event(PGFAULT);
+             count_memcg_event_mm(vma->vm_mm, PGFAULT);
Quoted 8 lines

There is one thing worth noticing is here vma may or may not be valid depending on the retval of the fault.

RETRY is exactly one of the cases that accessing vma may be unsafe due to releasing of mmap read lock. The other one is the recently added VM_FAULT_COMPLETE. So if we want to move this chunk (or any vma reference) to be later we need to consider a valid vma / mm being there first, or we're prone to accessing a vma that has already been released, I think.

Good catch! I think you are right and I should have stored vma->vm_mm in the beginning and used it when calling count_memcg_event_mm(). I'll prepare a new patch which handles this correctly. Thanks,
Suren.

Quoted 18 lines
Quoted 9 lines
mm/memory.c +1 -1
+     }
 
      return ret;
 }
--

2.40.0.634.g4ca3ef3211-goog

Thanks,

--
Peter Xu

--
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

author reviewer
#4 lore raw
Quoted 78 lines · Suren Baghdasaryan
Quoted 2 lines · Peter Xu

Hi, Suren,

Hi Peter,

Quoted 48 lines · Suren Baghdasaryan
Quoted 46 lines

If the page fault handler requests a retry, we will count the fault multiple times. This is a relatively harmless problem as the retry paths are not often requested, and the only user-visible problem is that the fault counter will be slightly higher than it should be. Nevertheless, userspace only took one fault, and should not see the fact that the kernel had to retry the fault multiple times.

Fixes 6b4c9f446981 ("filemap: drop the mmap_sem for all blocking operations")

---
Patch applies cleanly over linux-next and mm-unstable

 mm/memory.c | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)
mm/memory.c +5 -6
index 1c5b231fe6e3..d88f370eacd1 100644
@@ -5212,17 +5212,16 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
 
      __set_current_state(TASK_RUNNING);
 
-     count_vm_event(PGFAULT);
-     count_memcg_event_mm(vma->vm_mm, PGFAULT);
-
      ret = sanitize_fault_flags(vma, &flags);
      if (ret)
-             return ret;
+             goto out;
 
      if (!arch_vma_access_permitted(vma, flags & FAULT_FLAG_WRITE,
                                          flags & FAULT_FLAG_INSTRUCTION,
-                                         flags & FAULT_FLAG_REMOTE))
-             return VM_FAULT_SIGSEGV;
+                                         flags & FAULT_FLAG_REMOTE)) {
+             ret = VM_FAULT_SIGSEGV;
+             goto out;
+     }
 
      /*
       * Enable the memcg OOM handling for faults triggered in user
@@ -5253,6 +5252,11 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
      }
 
Quoted 4 lines

mm/memory.c vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, · line 5254

Quoted 1 line
mm/memory.c +0 -0
      mm_account_fault(regs, address, flags, ret);

Here is the mm_account_fault() function taking care of some other accountings. Perhaps good to put things into it?

That seems appropriate. Let me take a closer look.

Quoted 5 lines

It also already ignores invalid faults:

if (ret & (VM_FAULT_ERROR | VM_FAULT_RETRY))
return;

Can there be a case of (!VM_FAULT_ERROR && VM_FAULT_RETRY) - basically we need to retry but no errors happened? If so then this condition would double-count pagefaults in such cases. If such return code is impossible then it's the same as checking for VM_FAULT_RETRY.

Quoted 4 lines

I see that you may also want to account for sigbus, however I really don't know why. Explanations would be great when it would matter. So far it makes sense to me if we skip both RETRY or ERROR cases.

Quoted 6 lines

Accounting in case of a sigbus is not affected by this patch I think. We account for sigbus or any other error cases because there was a pagefault and we need to account for it. Whether we failed to handle it or not should not affect the count. We skip the retry case because we know the same fault will be retried. If we don't skip then we will double-count this fault.

mm_account_fault() has a nice comment explaining why it skips errors and that now makes sense to me. Let me move the accounting there and see if others agree that's the right place.

Quoted 40 lines
Quoted 7 lines

mm/memory.c vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, · line 5258

Quoted 6 lines
mm/memory.c +4 -0
+out:
+     if (!(ret & VM_FAULT_RETRY)) {
+             count_vm_event(PGFAULT);
+             count_memcg_event_mm(vma->vm_mm, PGFAULT);
Quoted 8 lines

There is one thing worth noticing is here vma may or may not be valid depending on the retval of the fault.

RETRY is exactly one of the cases that accessing vma may be unsafe due to releasing of mmap read lock. The other one is the recently added VM_FAULT_COMPLETE. So if we want to move this chunk (or any vma reference) to be later we need to consider a valid vma / mm being there first, or we're prone to accessing a vma that has already been released, I think.

Good catch! I think you are right and I should have stored vma->vm_mm in the beginning and used it when calling count_memcg_event_mm(). I'll prepare a new patch which handles this correctly. Thanks,
Suren.

Quoted 18 lines
Quoted 9 lines
mm/memory.c +1 -1
+     }
 
      return ret;
 }
--

2.40.0.634.g4ca3ef3211-goog

Thanks,

--
Peter Xu

--
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

#5 lore raw
In reply to Suren Baghdasaryan

Hi, Suren,

Quoted 8 lines · Suren Baghdasaryan
Quoted 4 lines

It also already ignores invalid faults:

if (ret & (VM_FAULT_ERROR | VM_FAULT_RETRY))
return;

Can there be a case of (!VM_FAULT_ERROR && VM_FAULT_RETRY) - basically we need to retry but no errors happened? If so then this condition would double-count pagefaults in such cases.

If ret==VM_FAULT_RETRY it should return here already, so I assume mm_account_fault() itself is fine regarding fault retries?

Note that I think "ret & (VM_FAULT_ERROR | VM_FAULT_RETRY)" above means "either ERROR or RETRY we'll skip the accounting".

IMHO we should have 3 cases here:

- ERROR && !RETRY
error triggered of any kind

- RETRY && !ERROR
we need to try one more time

- !RETRY && !ERROR
we finished the fault

I don't think ERROR & RETRY can even be set at the same time so I assume there's no option 4) - a RETRY should imply no ERROR already, even though it's still incomplete so need another attempt.

Thanks,

Signature · 1 line
Peter Xu
author reviewer
#6 lore raw
Quoted 21 lines · Peter Xu

Hi, Suren,

Quoted 8 lines · Suren Baghdasaryan
Quoted 4 lines

It also already ignores invalid faults:

if (ret & (VM_FAULT_ERROR | VM_FAULT_RETRY))
return;

Can there be a case of (!VM_FAULT_ERROR && VM_FAULT_RETRY) - basically we need to retry but no errors happened? If so then this condition would double-count pagefaults in such cases.

If ret==VM_FAULT_RETRY it should return here already, so I assume mm_account_fault() itself is fine regarding fault retries?

Note that I think "ret & (VM_FAULT_ERROR | VM_FAULT_RETRY)" above means "either ERROR or RETRY we'll skip the accounting".

IMHO we should have 3 cases here:

Quoted 8 lines

- ERROR && !RETRY
error triggered of any kind

- RETRY && !ERROR
we need to try one more time

- !RETRY && !ERROR
we finished the fault

After looking some more into mm_account_fault(), I think it would be fine to count the faults which produced errors. IIUC these counters represent the total number of faults, not the number of valid and successful faults. If so then I think simply using VM_FAULT_RETRY should be ok without considering all possible combinations. WDYT?

Quoted 10 lines

I don't think ERROR & RETRY can even be set at the same time so I assume there's no option 4) - a RETRY should imply no ERROR already, even though it's still incomplete so need another attempt.

Thanks,

--
Peter Xu

author reviewer
#7 lore raw
Quoted 29 lines · Suren Baghdasaryan
Quoted 27 lines · Peter Xu

Hi, Suren,

Quoted 8 lines · Suren Baghdasaryan
Quoted 4 lines

It also already ignores invalid faults:

if (ret & (VM_FAULT_ERROR | VM_FAULT_RETRY))
return;

Can there be a case of (!VM_FAULT_ERROR && VM_FAULT_RETRY) - basically we need to retry but no errors happened? If so then this condition would double-count pagefaults in such cases.

If ret==VM_FAULT_RETRY it should return here already, so I assume mm_account_fault() itself is fine regarding fault retries?

Note that I think "ret & (VM_FAULT_ERROR | VM_FAULT_RETRY)" above means "either ERROR or RETRY we'll skip the accounting".

IMHO we should have 3 cases here:

- ERROR && !RETRY
error triggered of any kind

- RETRY && !ERROR
we need to try one more time

Quoted 8 lines
Quoted 2 lines

- !RETRY && !ERROR
we finished the fault

After looking some more into mm_account_fault(), I think it would be fine to count the faults which produced errors. IIUC these counters represent the total number of faults, not the number of valid and successful faults. If so then I think simply using VM_FAULT_RETRY should be ok without considering all possible combinations. WDYT?

I posted v2 at https://lore.kernel.org/all/[email protected]/
Hopefully it's closer to what we want it to be.

Quoted 11 lines
Quoted 11 lines

I don't think ERROR & RETRY can even be set at the same time so I assume there's no option 4) - a RETRY should imply no ERROR already, even though it's still incomplete so need another attempt.

Thanks,

--
Peter Xu