Speculative page faults
Subsystems (4)
v11 cover Speculative page faults
Hi Laurent,
We are observing an issue with speculative page fault with the following test code on ARM64 (4.14 kernel, 8 cores).
Steps:
1) Run the program
2) The 2 threads will try lock/unlock and prints messages, and main thread does fork. In around 1 minute time, lock/unlock threads will hang on pthread_mutex_lock.
Initially the issue was reported when ptrace was tried on apps. Later it was discovered that the write protect done by fork is causing the issue and the below test code was created.
Observations:
1) We have tried disabling SPF and the issue disappears.
2) Adding this piece of code in __handle_speculative_fault also fixes the problem.
if (flags & FAULT_FLAG_WRITE && !pte_write(vmf.orig_pte))
return VM_FAULT_RETRY;
3) As an experiment we tried encapsulating handle_speculative_fault with down_read(mmap_sem) and that too fixes the problem.
4) It is observed that while in wp_page_copy, the contents of the old_page changes which should not ideally happen as the pte is !pte_write.
5) To prove that it is a race, we tried affining the threads to single core, and the issue disappears.
Let us know if you want us to try out any experiments.
Thanks,
Vinayak
/**test.c***/
#include <stdio.h>
#include <pthread.h>
#include <unistd.h>
#include <string.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <errno.h>
#define UNUSED(expr) do { (void)(expr); } while(0) #define print printf
pthread_mutex_t m;pthread_cond_t c;
static void* cond_do(void* arg)
{
UNUSED(arg);
do {
print("%s:%d state=%u addr: %lx\n", __func__, __LINE__, *(uint32_t*)(&m), (unsigned long)&m); pthread_mutex_lock(&m);
print("%s:%d state=%u addr: %lx\n", __func__, __LINE__, *(uint32_t*)(&m), (unsigned long)&m); pthread_mutex_unlock(&m);
print("%s:%d state=%u addr: %lx\n", __func__, __LINE__, *(uint32_t*)(&m), (unsigned long)&m); } while (true);
return NULL;
}
static void* sig_do(void* arg)
{
UNUSED(arg);
do {
print("%s:%d state=%u addr: %lx\n", __func__, __LINE__, *(uint32_t*)(&m), (unsigned long)&m); pthread_mutex_lock(&m);
print("%s:%d state=%u addr: %lx\n", __func__, __LINE__, *(uint32_t*)(&m), (unsigned long)&m); pthread_mutex_unlock(&m);
print("%s:%d state=%u addr: %lx\n", __func__, __LINE__, *(uint32_t*)(&m), (unsigned long)&m); } while (true);
return NULL;
}
int main()
{
pthread_t sig;
pthread_t cond;
pthread_mutex_init(&m, NULL);
pthread_create(&cond, NULL, cond_do, NULL);
pthread_create(&sig, NULL, sig_do, NULL);
while(1) {
if (!fork()) {
usleep(500);
abort();
}
usleep(550);
}
pthread_join(sig, NULL);
pthread_join(cond, NULL);
return 0;
}
Quoted 3 lines · Vinayak Menon
Hi Laurent,
We are observing an issue with speculative page fault with the following test code on ARM64 (4.14 kernel, 8 cores).
With the patch below, we don't hit the issue.
From: Vinayak Menon <[email protected]>
Date: Mon, 14 Jan 2019 16:06:34 +0530
Subject: [PATCH] mm: flush stale tlb entries on speculative write fault
It is observed that the following scenario results in threads A and B of process 1 blocking on pthread_mutex_lock forever after few iterations.
CPU 1 CPU 2 CPU 3 Process 1, Process 1, Process 1, Thread A Thread B Thread C
while (1) { while (1) { while(1) { pthread_mutex_lock(l) pthread_mutex_lock(l) fork pthread_mutex_unlock(l) pthread_mutex_unlock(l) } } }
When from thread C, copy_one_pte write-protects the parent pte (of lock l), stale tlb entries can exist with write permissions on one of the CPUs at least. This can create a problem if one of the threads A or B hits the write fault. Though dup_mmap calls flush_tlb_mm after copy_page_range, since speculative page fault does not take mmap_sem it can proceed further fixing a fault soon after CPU 3 does ptep_set_wrprotect. But the CPU with stale tlb entry can still modify old_page even after it is copied to new_page by wp_page_copy, thus causing a corruption.
---
mm/memory.c | 7 +++++++
1 file changed, 7 insertions(+)
mm/memory.c +0 -0
@@ -4507,6 +4507,13 @@ int __handle_speculative_fault(struct mm_struct *mm, unsigned long address,
return VM_FAULT_RETRY;
}
+ /*
+ * Discard tlb entries created before ptep_set_wrprotect + * in copy_one_pte
+ */
+ if (flags & FAULT_FLAG_WRITE && !pte_write(vmf.orig_pte)) + flush_tlb_page(vmf.vma, address);
+
mem_cgroup_oom_enable();
ret = handle_pte_fault(&vmf); mem_cgroup_oom_disable();
--
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, hosted by The Linux Foundation
Le 14/01/2019 à 14:19, Vinayak Menon a écrit :
Quoted 25 lines · Vinayak Menon
Quoted 3 lines
Hi Laurent,
We are observing an issue with speculative page fault with the following test code on ARM64 (4.14 kernel, 8 cores).
With the patch below, we don't hit the issue.
From: Vinayak Menon <[email protected]>
Date: Mon, 14 Jan 2019 16:06:34 +0530
Subject: [PATCH] mm: flush stale tlb entries on speculative write faultIt is observed that the following scenario results in threads A and B of process 1 blocking on
pthread_mutex_lockforever after few iterations.CPU 1 CPU 2 CPU 3 Process 1, Process 1, Process 1, Thread A Thread B Thread C
while (1) { while (1) { while(1) { pthread_mutex_lock(l) pthread_mutex_lock(l) fork
pthread_mutex_unlock(l)pthread_mutex_unlock(l) } } }
Quoted 9 lines
When from thread C,
copy_one_ptewrite-protects the parent pte (of lock l), stale tlb entries can exist with write permissions on one of the CPUs at least. This can create a problem if one of the threads A or B hits the write fault. Thoughdup_mmapcallsflush_tlb_mmaftercopy_page_range, since speculative page fault does not takemmap_semit can proceed further fixing a fault soon after CPU 3 doesptep_set_wrprotect. But the CPU with stale tlb entry can still modifyold_pageeven after it is copied tonew_pagebywp_page_copy, thus causing a corruption.
Nice catch and thanks for your investigation!
There is a real synchronization issue here between copy_page_range() and the speculative page fault handler. I didn't get it on PowerVM since the TLB are flushed when arch_exit_lazy_mode() is called in copy_page_range() but now, I can get it when running on x86_64.
Quoted 13 lines
Signed-off-by Vinayak Menon <[email protected]>--- mm/memory.c | 7 +++++++ 1 file changed, 7 insertions(+)mm/memory.c +0
-0@@ -4507,6 +4507,13 @@ int __handle_speculative_fault(struct mm_struct *mm, unsigned long address, return VM_FAULT_RETRY; }
mm/memory.c int __handle_speculative_fault(struct mm_struct *mm, unsigned long address, · line 4519
Quoted 10 lines
mm/memory.c +7
-0+ /* + * Discard tlb entries created before ptep_set_wrprotect + * in copy_one_pte + */ + if (flags & FAULT_FLAG_WRITE && !pte_write(vmf.orig_pte)) + flush_tlb_page(vmf.vma, address); + mem_cgroup_oom_enable(); ret = handle_pte_fault(&vmf); mem_cgroup_oom_disable();
Your patch is fixing the race but I'm wondering about the cost of these tlb flushes. Here we are flushing on a per page basis (architecture like x86_64 are smarter and flush more pages) but there is a request to flush a range of tlb entries each time a cow page is newly touched. I think there could be some bad impact here.
Another option would be to flush the range in copy_pte_range() before unlocking the page table lock. This will flush entries flush_tlb_mm() would later handle in dup_mmap() but that will be called once per fork per cow VMA.
I tried the attached patch which seems to fix the issue on x86_64. Could you please give it a try on arm64 ?
Thanks,
Laurent.
From 9847338187c5c7e2d387d14765452d00fa60981e Mon Sep 17 00:00:00 2001
From: Laurent Dufour <[email protected]>
Date: Mon, 14 Jan 2019 18:35:39 +0100
Subject: [PATCH] mm: flush TLB once pages are copied when SPF is on
Vinayak Menon reported that the following scenario results in threads A and B of process 1 blocking on pthread_mutex_lock forever after few iterations.
CPU 1 CPU 2 CPU 3 Process 1, Process 1, Process 1, Thread A Thread B Thread C
while (1) { while (1) { while(1) {
pthread_mutex_lock(l) pthread_mutex_lock(l) fork
pthread_mutex_unlock(l) pthread_mutex_unlock(l) }} }
When from thread C, copy_one_pte write-protects the parent pte (of lock l), stale tlb entries can exist with write permissions on one of the CPUs at least. This can create a problem if one of the threads A or B hits the write fault. Though dup_mmap calls flush_tlb_mm after copy_page_range, since speculative page fault does not take mmap_sem it can proceed further fixing a fault soon after CPU 3 does ptep_set_wrprotect. But the CPU with stale tlb entry can still modify old_page even after it is copied to new_page by wp_page_copy, thus causing a corruption.
--- mm/memory.c | 9 +++++++++ 1 file changed, 9 insertions(+)
mm/memory.c +9 -0
@@ -1112,6 +1112,15 @@ static int copy_pte_range(struct mm_struct *dst_mm, struct mm_struct *src_mm, }while (dst_pte++, src_pte++, addr +=PAGE_SIZE , addr != end);arch_leave_lazy_mmu_mode (); + +/* +* Prevent the page fault handler to copy the page while stale tlb entry +* are still not flushed. +*/ +if (IS_ENABLED (CONFIG_SPECULATIVE_PAGE_FAULT ) && +is_cow_mapping (vma->vm_flags)) +flush_tlb_range (vma, addr, end); +spin_unlock (src_ptl);pte_unmap (orig_src_pte);add_mm_rss_vec (dst_mm, rss);
Signature · 1 line
2.20.1
Quoted 66 lines · Laurent Dufour
Le 14/01/2019 à 14:19, Vinayak Menon a écrit :
Quoted 25 lines · Vinayak Menon
Quoted 3 lines
Hi Laurent,
We are observing an issue with speculative page fault with the following test code on ARM64 (4.14 kernel, 8 cores).
With the patch below, we don't hit the issue.
From: Vinayak Menon <[email protected]>
Date: Mon, 14 Jan 2019 16:06:34 +0530
Subject: [PATCH] mm: flush stale tlb entries on speculative write faultIt is observed that the following scenario results in threads A and B of process 1 blocking on
pthread_mutex_lockforever after few iterations.CPU 1 CPU 2 CPU 3 Process 1, Process 1, Process 1, Thread A Thread B Thread C
while (1) { while (1) { while(1) { pthread_mutex_lock(l) pthread_mutex_lock(l) fork
pthread_mutex_unlock(l)pthread_mutex_unlock(l) } } }Quoted 9 lines
When from thread C,
copy_one_ptewrite-protects the parent pte (of lock l), stale tlb entries can exist with write permissions on one of the CPUs at least. This can create a problem if one of the threads A or B hits the write fault. Thoughdup_mmapcallsflush_tlb_mmaftercopy_page_range, since speculative page fault does not takemmap_semit can proceed further fixing a fault soon after CPU 3 doesptep_set_wrprotect. But the CPU with stale tlb entry can still modifyold_pageeven after it is copied tonew_pagebywp_page_copy, thus causing a corruption.Nice catch and thanks for your investigation!
There is a real synchronization issue here between
copy_page_range()and the speculative page fault handler. I didn't get it on PowerVM since the TLB are flushed whenarch_exit_lazy_mode()is called incopy_page_range()but now, I can get it when running onx86_64.Quoted 13 lines
Signed-off-by Vinayak Menon <[email protected]>---
mm/memory.c | 7 +++++++1 file changed, 7 insertions(+)mm/memory.c +0
-0@@ -4507,6 +4507,13 @@ int __handle_speculative_fault(struct mm_struct *mm, unsigned long address, return VM_FAULT_RETRY; }mm/memory.c int __handle_speculative_fault(struct mm_struct *mm, unsigned long address, · line 4519
Quoted 10 lines
mm/memory.c +7
-0+ /* + * Discard tlb entries created before ptep_set_wrprotect + * in copy_one_pte + */ + if (flags & FAULT_FLAG_WRITE && !pte_write(vmf.orig_pte)) + flush_tlb_page(vmf.vma, address); + mem_cgroup_oom_enable(); ret = handle_pte_fault(&vmf); mem_cgroup_oom_disable();Your patch is fixing the race but I'm wondering about the cost of these tlb flushes. Here we are flushing on a per page basis (architecture like
x86_64are smarter and flush more pages) but there is a request to flush a range of tlb entries each time a cow page is newly touched. I think there could be some bad impact here.
Quoted 1 line
Another option would be to flush the range in
copy_pte_range()before unlocking the page table lock. This will flush entriesflush_tlb_mm()would later handle indup_mmap()but that will be called once per fork per cow VMA.
But wouldn't this cause an unnecessary impact if most of the COW pages remain untouched (which I assume would be the usual case) and thus do not create a fault ?
Quoted 3 lines
I tried the attached patch which seems to fix the issue on
x86_64. Could you please give it a try on arm64 ?
Your patch works fine on arm64 with a minor change. Thanks Laurent.
mm/memory.c +0 -0
@@ -1087,6 +1087,7 @@ static int copy_pte_range(struct mm_struct *dst_mm, struct mm_struct *src_mm,
spinlock_t *src_ptl, *dst_ptl;
int progress = 0;
int rss[NR_MM_COUNTERS];
+ unsigned long orig_addr = addr;
swp_entry_t entry = (swp_entry_t){0};
again:
@@ -1125,6 +1126,15 @@ static int copy_pte_range(struct mm_struct *dst_mm, struct mm_struct *src_mm, } while (dst_pte++, src_pte++, addr += PAGE_SIZE, addr != end);
arch_leave_lazy_mmu_mode();
+
+ /*
+ * Prevent the page fault handler to copy the page while stale tlb entry + * are still not flushed.
+ */
+ if (IS_ENABLED(CONFIG_SPECULATIVE_PAGE_FAULT) &&
+ is_cow_mapping(vma->vm_flags))
+ flush_tlb_range(vma, orig_addr, end);
+
spin_unlock(src_ptl);
pte_unmap(orig_src_pte);
add_mm_rss_vec(dst_mm, rss);
Thanks,
Vinayak
Le 16/01/2019 à 12:41, Vinayak Menon a écrit :
Quoted 68 lines · Laurent Dufour
Quoted 66 lines
Le 14/01/2019 à 14:19, Vinayak Menon a écrit :
Quoted 25 lines · Vinayak Menon
Quoted 3 lines
Hi Laurent,
We are observing an issue with speculative page fault with the following test code on ARM64 (4.14 kernel, 8 cores).
With the patch below, we don't hit the issue.
From: Vinayak Menon <[email protected]>
Date: Mon, 14 Jan 2019 16:06:34 +0530
Subject: [PATCH] mm: flush stale tlb entries on speculative write faultIt is observed that the following scenario results in threads A and B of process 1 blocking on
pthread_mutex_lockforever after few iterations.CPU 1 CPU 2 CPU 3 Process 1, Process 1, Process 1, Thread A Thread B Thread C
while (1) { while (1) { while(1) { pthread_mutex_lock(l) pthread_mutex_lock(l) fork
pthread_mutex_unlock(l)pthread_mutex_unlock(l) } } }Quoted 9 lines
When from thread C,
copy_one_ptewrite-protects the parent pte (of lock l), stale tlb entries can exist with write permissions on one of the CPUs at least. This can create a problem if one of the threads A or B hits the write fault. Thoughdup_mmapcallsflush_tlb_mmaftercopy_page_range, since speculative page fault does not takemmap_semit can proceed further fixing a fault soon after CPU 3 doesptep_set_wrprotect. But the CPU with stale tlb entry can still modifyold_pageeven after it is copied tonew_pagebywp_page_copy, thus causing a corruption.Nice catch and thanks for your investigation!
There is a real synchronization issue here between
copy_page_range()and the speculative page fault handler. I didn't get it on PowerVM since the TLB are flushed whenarch_exit_lazy_mode()is called incopy_page_range()but now, I can get it when running onx86_64.Quoted 13 lines
Signed-off-by Vinayak Menon <[email protected]>--- mm/memory.c | 7 +++++++ 1 file changed, 7 insertions(+)mm/memory.c +0
-0@@ -4507,6 +4507,13 @@ int __handle_speculative_fault(struct mm_struct *mm, unsigned long address, return VM_FAULT_RETRY; }mm/memory.c int __handle_speculative_fault(struct mm_struct *mm, unsigned long address, · line 4519
Quoted 10 lines
mm/memory.c +7
-0+ /* + * Discard tlb entries created before ptep_set_wrprotect + * in copy_one_pte + */ + if (flags & FAULT_FLAG_WRITE && !pte_write(vmf.orig_pte)) + flush_tlb_page(vmf.vma, address); + mem_cgroup_oom_enable(); ret = handle_pte_fault(&vmf); mem_cgroup_oom_disable();Your patch is fixing the race but I'm wondering about the cost of these tlb flushes. Here we are flushing on a per page basis (architecture like
x86_64are smarter and flush more pages) but there is a request to flush a range of tlb entries each time a cow page is newly touched. I think there could be some bad impact here.
Quoted 4 lines
Quoted 1 line
Another option would be to flush the range in
copy_pte_range()before unlocking the page table lock. This will flush entriesflush_tlb_mm()would later handle indup_mmap()but that will be called once per fork per cow VMA.But wouldn't this cause an unnecessary impact if most of the COW pages remain untouched (which I assume would be the usual case) and thus do not create a fault ?
I think this should be less costly to do it per vma at the time of the fork instead of per page hit once the fork has been done, since this will happen in both the forked task and the forking one (the COW pages are concerning the both sides of the fork).
Quoted 5 lines
Quoted 3 lines
I tried the attached patch which seems to fix the issue on
x86_64. Could you please give it a try on arm64 ?Your patch works fine on arm64 with a minor change. Thanks Laurent.
Yup my mistake !
I tried to shrink the patch after testing it, sounds that I shrunk it far too much...
Quoted 34 lines
mm/memory.c +10
-0@@ -1087,6 +1087,7 @@ static int copy_pte_range(struct mm_struct *dst_mm, struct mm_struct *src_mm, spinlock_t *src_ptl, *dst_ptl; int progress = 0; int rss[NR_MM_COUNTERS]; + unsigned long orig_addr = addr; swp_entry_t entry = (swp_entry_t){0}; again:@@ -1125,6 +1126,15 @@ static int copy_pte_range(struct mm_struct *dst_mm, struct mm_struct *src_mm, } while (dst_pte++, src_pte++, addr += PAGE_SIZE, addr != end); arch_leave_lazy_mmu_mode(); + + /* + * Prevent the page fault handler to copy the page while stale tlb entry + * are still not flushed. + */ + if (IS_ENABLED(CONFIG_SPECULATIVE_PAGE_FAULT) && + is_cow_mapping(vma->vm_flags)) + flush_tlb_range(vma, orig_addr, end); + spin_unlock(src_ptl); pte_unmap(orig_src_pte); add_mm_rss_vec(dst_mm, rss);Thanks,
Vinayak
Quoted 66 lines · Laurent Dufour
Le 14/01/2019 à 14:19, Vinayak Menon a écrit :
Quoted 25 lines · Vinayak Menon
Quoted 3 lines
Hi Laurent,
We are observing an issue with speculative page fault with the following test code on ARM64 (4.14 kernel, 8 cores).
With the patch below, we don't hit the issue.
From: Vinayak Menon <[email protected]>
Date: Mon, 14 Jan 2019 16:06:34 +0530
Subject: [PATCH] mm: flush stale tlb entries on speculative write faultIt is observed that the following scenario results in threads A and B of process 1 blocking on
pthread_mutex_lockforever after few iterations.CPU 1 CPU 2 CPU 3 Process 1, Process 1, Process 1, Thread A Thread B Thread C
while (1) { while (1) { while(1) { pthread_mutex_lock(l) pthread_mutex_lock(l) fork
pthread_mutex_unlock(l)pthread_mutex_unlock(l) } } }Quoted 9 lines
When from thread C,
copy_one_ptewrite-protects the parent pte (of lock l), stale tlb entries can exist with write permissions on one of the CPUs at least. This can create a problem if one of the threads A or B hits the write fault. Thoughdup_mmapcallsflush_tlb_mmaftercopy_page_range, since speculative page fault does not takemmap_semit can proceed further fixing a fault soon after CPU 3 doesptep_set_wrprotect. But the CPU with stale tlb entry can still modifyold_pageeven after it is copied tonew_pagebywp_page_copy, thus causing a corruption.Nice catch and thanks for your investigation!
There is a real synchronization issue here between
copy_page_range()and the speculative page fault handler. I didn't get it on PowerVM since the TLB are flushed whenarch_exit_lazy_mode()is called incopy_page_range()but now, I can get it when running onx86_64.Quoted 13 lines
Signed-off-by Vinayak Menon <[email protected]>---
mm/memory.c | 7 +++++++1 file changed, 7 insertions(+)mm/memory.c +0
-0@@ -4507,6 +4507,13 @@ int __handle_speculative_fault(struct mm_struct *mm, unsigned long address, return VM_FAULT_RETRY; }mm/memory.c int __handle_speculative_fault(struct mm_struct *mm, unsigned long address, · line 4519
Quoted 10 lines
mm/memory.c +7
-0+ /* + * Discard tlb entries created before ptep_set_wrprotect + * in copy_one_pte + */ + if (flags & FAULT_FLAG_WRITE && !pte_write(vmf.orig_pte)) + flush_tlb_page(vmf.vma, address); + mem_cgroup_oom_enable(); ret = handle_pte_fault(&vmf); mem_cgroup_oom_disable();Your patch is fixing the race but I'm wondering about the cost of these tlb flushes. Here we are flushing on a per page basis (architecture like
x86_64are smarter and flush more pages) but there is a request to flush a range of tlb entries each time a cow page is newly touched. I think there could be some bad impact here.
Quoted 1 line
Another option would be to flush the range in
copy_pte_range()before unlocking the page table lock. This will flush entriesflush_tlb_mm()would later handle indup_mmap()but that will be called once per fork per cow VMA.
But wouldn't this cause an unnecessary impact if most of the COW pages remain untouched (which I assume would be the usual case) and thus do not create a fault ?
Quoted 3 lines
I tried the attached patch which seems to fix the issue on
x86_64. Could you please give it a try on arm64 ?
Your patch works fine on arm64 with a minor change. Thanks Laurent.
mm/memory.c +0 -0
@@ -1087,6 +1087,7 @@ static int copy_pte_range(struct mm_struct *dst_mm, struct mm_struct *src_mm,
spinlock_t *src_ptl, *dst_ptl;
int progress = 0;
int rss[NR_MM_COUNTERS];
+ unsigned long orig_addr = addr;
swp_entry_t entry = (swp_entry_t){0};
again:
@@ -1125,6 +1126,15 @@ static int copy_pte_range(struct mm_struct *dst_mm, struct mm_struct *src_mm, } while (dst_pte++, src_pte++, addr += PAGE_SIZE, addr != end);
arch_leave_lazy_mmu_mode();
+
+ /*
+ * Prevent the page fault handler to copy the page while stale tlb entry + * are still not flushed.
+ */
+ if (IS_ENABLED(CONFIG_SPECULATIVE_PAGE_FAULT) &&
+ is_cow_mapping(vma->vm_flags))
+ flush_tlb_range(vma, orig_addr, end);
+
spin_unlock(src_ptl);
pte_unmap(orig_src_pte);
add_mm_rss_vec(dst_mm, rss);
Thanks,
Vinayak
Quoted 64 lines · Vinayak Menon
Quoted 63 lines · Laurent Dufour
Quoted 24 lines · Vinayak Menon
Quoted 3 lines · Vinayak Menon
Hi Laurent,
We are observing an issue with speculative page fault with the following test code on ARM64 (4.14 kernel, 8 cores).
With the patch below, we don't hit the issue.
From: Vinayak Menon <[email protected]>
Date: Mon, 14 Jan 2019 16:06:34 +0530
Subject: [PATCH] mm: flush stale tlb entries on speculative write faultIt is observed that the following scenario results in threads A and B of process 1 blocking on
pthread_mutex_lockforever after few iterations.CPU 1 CPU 2 CPU 3 Process 1, Process 1, Process 1, Thread A Thread B Thread Cwhile (1) { while (1) { while(1) { pthread_mutex_lock(l) pthread_mutex_lock(l) fork pthread_mutex_unlock(l) pthread_mutex_unlock(l) }} }
Quoted 9 lines
When from thread C,
copy_one_ptewrite-protects the parent pte (of lock l), stale tlb entries can exist with write permissions on one of the CPUs at least. This can create a problem if one of the threads A or B hits the write fault. Thoughdup_mmapcallsflush_tlb_mmaftercopy_page_range, since speculative page fault does not takemmap_semit can proceed further fixing a fault soon after CPU 3 doesptep_set_wrprotect. But the CPU with stale tlb entry can still modifyold_pageeven after it is copied tonew_pagebywp_page_copy, thus causing a corruption.Nice catch and thanks for your investigation!
There is a real synchronization issue here between
copy_page_range()and the speculative page fault handler. I didn't get it on PowerVM since the TLB are flushed whenarch_exit_lazy_mode()is called incopy_page_range()but now, I can get it when running onx86_64.Quoted 13 lines
Signed-off-by Vinayak Menon <[email protected]>--- mm/memory.c | 7 +++++++ 1 file changed, 7 insertions(+)mm/memory.c +0
-0@@ -4507,6 +4507,13 @@ int __handle_speculative_fault(struct mm_struct *mm, unsigned long address,return VM_FAULT_RETRY ; }mm/memory.c int __handle_speculative_fault(struct mm_struct *mm, unsigned long address, · line 4519
Quoted 10 lines
mm/memory.c +7
-0+/* +* Discard tlb entries created before ptep_set_wrprotect +* in copy_one_pte +*/ +if (flags &FAULT_FLAG_WRITE && !pte_write (vmf.orig_pte)) +flush_tlb_page (vmf.vma, address); +mem_cgroup_oom_enable (); ret =handle_pte_fault (&vmf);mem_cgroup_oom_disable ();Your patch is fixing the race but I'm wondering about the cost of these tlb flushes. Here we are flushing on a per page basis (architecture like
x86_64are smarter and flush more pages) but there is a request to flush a range of tlb entries each time a cow page is newly touched. I think there could be some bad impact here.
Quoted 8 lines
Quoted 1 line
Another option would be to flush the range in
copy_pte_range()before unlocking the page table lock. This will flush entriesflush_tlb_mm()would later handle indup_mmap()but that will be called once per fork per cow VMA.But wouldn't this cause an unnecessary impact if most of the COW pages remain untouched (which I assume would be the usual case) and thus do not create a fault ?
Quoted 2 lines
I tried the attached patch which seems to fix the issue on
x86_64. Could you please give it a try on arm64 ?Your patch works fine on arm64 with a minor change. Thanks Laurent.
Hi, Vinayak and Laurent
I think the below change will impact the performance significantly. Becuase most of process has many vmas with cow flags. Flush the tlb in advance is not the better way to avoid the issue and it will call the flush_tlb_mm later.
I think we can try the following way to do.
vm_write_begin(vma)copy_pte_rangevm_write_end(vma)
The speculative page fault will return to grap the mmap_sem to run the nromal path. Any thought?
Thanks,
zhong jiang
Quoted 35 lines
mm/memory.c +10
-0@@ -1087,6 +1087,7 @@ static int copy_pte_range(struct mm_struct *dst_mm, struct mm_struct *src_mm,spinlock_t *src_ptl, *dst_ptl;int progress =0 ;int rss[NR_MM_COUNTERS ]; +unsigned long orig_addr = addr;swp_entry_t entry = (swp_entry_t ){0 }; again:@@ -1125,6 +1126,15 @@ static int copy_pte_range(struct mm_struct *dst_mm, struct mm_struct *src_mm, }while (dst_pte++, src_pte++, addr +=PAGE_SIZE , addr != end);arch_leave_lazy_mmu_mode (); + +/* +* Prevent the page fault handler to copy the page while stale tlb entry +* are still not flushed. +*/ +if (IS_ENABLED (CONFIG_SPECULATIVE_PAGE_FAULT ) && +is_cow_mapping (vma->vm_flags)) +flush_tlb_range (vma, orig_addr, end); +spin_unlock (src_ptl);pte_unmap (orig_src_pte);add_mm_rss_vec (dst_mm, rss);Thanks,
Vinayak
Le 17/01/2019 à 16:51, zhong jiang a écrit :
Quoted 79 lines · Vinayak Menon
Quoted 64 lines · Laurent Dufour
Quoted 63 lines · Vinayak Menon
Quoted 24 lines · Vinayak Menon
Quoted 3 lines
Hi Laurent,
We are observing an issue with speculative page fault with the following test code on ARM64 (4.14 kernel, 8 cores).
With the patch below, we don't hit the issue.
From: Vinayak Menon <[email protected]>
Date: Mon, 14 Jan 2019 16:06:34 +0530
Subject: [PATCH] mm: flush stale tlb entries on speculative write faultIt is observed that the following scenario results in threads A and B of process 1 blocking on
pthread_mutex_lockforever after few iterations.CPU 1 CPU 2 CPU 3 Process 1, Process 1, Process 1, Thread A Thread B Thread Cwhile (1) { while (1) { while(1) { pthread_mutex_lock(l) pthread_mutex_lock(l) fork pthread_mutex_unlock(l) pthread_mutex_unlock(l) }} }
Quoted 9 lines
When from thread C,
copy_one_ptewrite-protects the parent pte (of lock l), stale tlb entries can exist with write permissions on one of the CPUs at least. This can create a problem if one of the threads A or B hits the write fault. Thoughdup_mmapcallsflush_tlb_mmaftercopy_page_range, since speculative page fault does not takemmap_semit can proceed further fixing a fault soon after CPU 3 doesptep_set_wrprotect. But the CPU with stale tlb entry can still modifyold_pageeven after it is copied tonew_pagebywp_page_copy, thus causing a corruption.Nice catch and thanks for your investigation!
There is a real synchronization issue here between
copy_page_range()and the speculative page fault handler. I didn't get it on PowerVM since the TLB are flushed whenarch_exit_lazy_mode()is called incopy_page_range()but now, I can get it when running onx86_64.Quoted 13 lines
Signed-off-by Vinayak Menon <[email protected]>--- mm/memory.c | 7 +++++++ 1 file changed, 7 insertions(+)mm/memory.c +0
-0@@ -4507,6 +4507,13 @@ int __handle_speculative_fault(struct mm_struct *mm, unsigned long address,return VM_FAULT_RETRY ; }mm/memory.c int __handle_speculative_fault(struct mm_struct *mm, unsigned long address, · line 4519
Quoted 10 lines
mm/memory.c +7
-0+/* +* Discard tlb entries created before ptep_set_wrprotect +* in copy_one_pte +*/ +if (flags &FAULT_FLAG_WRITE && !pte_write (vmf.orig_pte)) +flush_tlb_page (vmf.vma, address); +mem_cgroup_oom_enable (); ret =handle_pte_fault (&vmf);mem_cgroup_oom_disable ();Your patch is fixing the race but I'm wondering about the cost of these tlb flushes. Here we are flushing on a per page basis (architecture like
x86_64are smarter and flush more pages) but there is a request to flush a range of tlb entries each time a cow page is newly touched. I think there could be some bad impact here.Quoted 8 lines
Quoted 1 line
Another option would be to flush the range in
copy_pte_range()before unlocking the page table lock. This will flush entriesflush_tlb_mm()would later handle indup_mmap()but that will be called once per fork per cow VMA.But wouldn't this cause an unnecessary impact if most of the COW pages remain untouched (which I assume would be the usual case) and thus do not create a fault ?
Quoted 2 lines
I tried the attached patch which seems to fix the issue on
x86_64. Could you please give it a try on arm64 ?Your patch works fine on arm64 with a minor change. Thanks Laurent.
Hi, Vinayak and Laurent
I think the below change will impact the performance significantly. Becuase most of process has many vmas with cow flags. Flush the tlb in advance is not the better way to avoid the issue and it will call the
flush_tlb_mmlater.
Quoted 8 lines
I think we can try the following way to do.
vm_write_begin(vma)copy_pte_rangevm_write_end(vma)The speculative page fault will return to grap the
mmap_semto run the nromal path. Any thought?
Hi Zhong,
I agree that flushing the TLB could have a bad impact on the performance, but tagging the VMA when copy_pte_range() is not fixing the issue as the VMA must be flagged until the PTE are flushed.
Here is what happens:
CPU A CPU B CPU Cfork()copy_pte_range()
set PTE rdonly
got to next VMA...
. PTE is seen rdonly PTE still writable . thread is writing to page . -> page fault
. copy the page Thread writes to page
. . -> no page fault
. update the PTE
. flush TLB for that PTE
flush TLB PTE are now rdonly
So the write done by the CPU C is interfering with the page copy operation done by CPU B, leading to the data corruption.
Flushing the PTE in copy_pte_range() is fixing the issue as the CPU C is seeing the PTE as rdonly earlier. But this impacts performance.
Another option, I'll work on is to flag _all the COW eligible_ VMA before starting copying them and until the PTE are flushed on the CPU A. This way when the CPU B will page fault the speculative handler will abort because the VMA is in the way to be touched.
But I need to ensure that all the calls to copy_pte_range() are handling this correctly.
Laurent.
Quoted 103 lines · Laurent Dufour
Quoted 79 lines · zhong jiang
Quoted 64 lines · Vinayak Menon
Quoted 63 lines · Laurent Dufour
Quoted 24 lines · Vinayak Menon
Quoted 3 lines · Vinayak Menon
Hi Laurent,
We are observing an issue with speculative page fault with the following test code on ARM64 (4.14 kernel, 8 cores).
With the patch below, we don't hit the issue.
From: Vinayak Menon <[email protected]>
Date: Mon, 14 Jan 2019 16:06:34 +0530
Subject: [PATCH] mm: flush stale tlb entries on speculative write faultIt is observed that the following scenario results in threads A and B of process 1 blocking on
pthread_mutex_lockforever after few iterations.CPU 1 CPU 2 CPU 3 Process 1, Process 1, Process 1, Thread A Thread B Thread Cwhile (1) { while (1) { while(1) { pthread_mutex_lock(l) pthread_mutex_lock(l) fork pthread_mutex_unlock(l) pthread_mutex_unlock(l) }} }
Quoted 9 lines
When from thread C,
copy_one_ptewrite-protects the parent pte (of lock l), stale tlb entries can exist with write permissions on one of the CPUs at least. This can create a problem if one of the threads A or B hits the write fault. Thoughdup_mmapcallsflush_tlb_mmaftercopy_page_range, since speculative page fault does not takemmap_semit can proceed further fixing a fault soon after CPU 3 doesptep_set_wrprotect. But the CPU with stale tlb entry can still modifyold_pageeven after it is copied tonew_pagebywp_page_copy, thus causing a corruption.Nice catch and thanks for your investigation!
There is a real synchronization issue here between
copy_page_range()and the speculative page fault handler. I didn't get it on PowerVM since the TLB are flushed whenarch_exit_lazy_mode()is called incopy_page_range()but now, I can get it when running onx86_64.Quoted 13 lines
Signed-off-by Vinayak Menon <[email protected]>--- mm/memory.c | 7 +++++++ 1 file changed, 7 insertions(+)mm/memory.c +0
-0@@ -4507,6 +4507,13 @@ int __handle_speculative_fault(struct mm_struct *mm, unsigned long address,return VM_FAULT_RETRY ; }mm/memory.c int __handle_speculative_fault(struct mm_struct *mm, unsigned long address, · line 4519
Quoted 10 lines
mm/memory.c +7
-0+/* +* Discard tlb entries created before ptep_set_wrprotect +* in copy_one_pte +*/ +if (flags &FAULT_FLAG_WRITE && !pte_write (vmf.orig_pte)) +flush_tlb_page (vmf.vma, address); +mem_cgroup_oom_enable (); ret =handle_pte_fault (&vmf);mem_cgroup_oom_disable ();Your patch is fixing the race but I'm wondering about the cost of these tlb flushes. Here we are flushing on a per page basis (architecture like
x86_64are smarter and flush more pages) but there is a request to flush a range of tlb entries each time a cow page is newly touched. I think there could be some bad impact here.Quoted 8 lines
Quoted 1 line
Another option would be to flush the range in
copy_pte_range()before unlocking the page table lock. This will flush entriesflush_tlb_mm()would later handle indup_mmap()but that will be called once per fork per cow VMA.But wouldn't this cause an unnecessary impact if most of the COW pages remain untouched (which I assume would be the usual case) and thus do not create a fault ?
Quoted 2 lines
I tried the attached patch which seems to fix the issue on
x86_64. Could you please give it a try on arm64 ?Your patch works fine on arm64 with a minor change. Thanks Laurent.
Hi, Vinayak and Laurent
I think the below change will impact the performance significantly. Becuase most of process has many vmas with cow flags. Flush the tlb in advance is not the better way to avoid the issue and it will call the
flush_tlb_mmlater.Quoted 8 lines
I think we can try the following way to do.
vm_write_begin(vma)copy_pte_rangevm_write_end(vma)The speculative page fault will return to grap the
mmap_semto run the nromal path. Any thought?Hi Zhong,
I agree that flushing the TLB could have a bad impact on the performance, but tagging the VMA when
copy_pte_range()is not fixing the issue as the VMA must be flagged until the PTE are flushed.Here is what happens:
CPU A CPU B CPU C
fork()copy_pte_range()
set PTE rdonly
got to next VMA.... PTE is seen rdonly PTE still writable . thread is writing to page . -> page fault
Quoted 8 lines
. copy the page Thread writes to page
. . -> no page fault
. update the PTE
. flush TLB for that PTE
flush TLB PTE are now rdonlySo the write done by the CPU C is interfering with the page copy operation done by CPU B, leading to the data corruption.
I want to know the case if the CPU B has finished in front of the CPU C that the data still is vaild ?
This is to say, the old_page will be changed from other cpu because of the access from other cpu.
Maybe this is a stupid qestion :-)
Thanks,
zhong jiang.
Quoted 12 lines
Flushing the PTE in
copy_pte_range()is fixing the issue as the CPU C is seeing the PTE as rdonly earlier. But this impacts performance.Another option, I'll work on is to flag
_allthe COWeligible_VMA before starting copying them and until the PTE are flushed on the CPU A. This way when the CPU B will page fault the speculative handler will abort because the VMA is in the way to be touched.But I need to ensure that all the calls to
copy_pte_range()are handling this correctly.Laurent.
.
Le 18/01/2019 à 16:41, zhong jiang a écrit :
Quoted 110 lines · Laurent Dufour
Quoted 109 lines · zhong jiang
Quoted 79 lines · Vinayak Menon
Quoted 64 lines · Laurent Dufour
Quoted 63 lines · Vinayak Menon
Quoted 24 lines · Vinayak Menon
Quoted 3 lines
Hi Laurent,
We are observing an issue with speculative page fault with the following test code on ARM64 (4.14 kernel, 8 cores).
With the patch below, we don't hit the issue.
From: Vinayak Menon <[email protected]>
Date: Mon, 14 Jan 2019 16:06:34 +0530
Subject: [PATCH] mm: flush stale tlb entries on speculative write faultIt is observed that the following scenario results in threads A and B of process 1 blocking on
pthread_mutex_lockforever after few iterations.CPU 1 CPU 2 CPU 3 Process 1, Process 1, Process 1, Thread A Thread B Thread Cwhile (1) { while (1) { while(1) { pthread_mutex_lock(l) pthread_mutex_lock(l) fork pthread_mutex_unlock(l) pthread_mutex_unlock(l) }} }
Quoted 9 lines
When from thread C,
copy_one_ptewrite-protects the parent pte (of lock l), stale tlb entries can exist with write permissions on one of the CPUs at least. This can create a problem if one of the threads A or B hits the write fault. Thoughdup_mmapcallsflush_tlb_mmaftercopy_page_range, since speculative page fault does not takemmap_semit can proceed further fixing a fault soon after CPU 3 doesptep_set_wrprotect. But the CPU with stale tlb entry can still modifyold_pageeven after it is copied tonew_pagebywp_page_copy, thus causing a corruption.Nice catch and thanks for your investigation!
There is a real synchronization issue here between
copy_page_range()and the speculative page fault handler. I didn't get it on PowerVM since the TLB are flushed whenarch_exit_lazy_mode()is called incopy_page_range()but now, I can get it when running onx86_64.Quoted 13 lines
Signed-off-by Vinayak Menon <[email protected]>--- mm/memory.c | 7 +++++++ 1 file changed, 7 insertions(+)mm/memory.c +0
-0@@ -4507,6 +4507,13 @@ int __handle_speculative_fault(struct mm_struct *mm, unsigned long address,return VM_FAULT_RETRY ; }mm/memory.c int __handle_speculative_fault(struct mm_struct *mm, unsigned long address, · line 4519
Quoted 10 lines
mm/memory.c +7
-0+/* +* Discard tlb entries created before ptep_set_wrprotect +* in copy_one_pte +*/ +if (flags &FAULT_FLAG_WRITE && !pte_write (vmf.orig_pte)) +flush_tlb_page (vmf.vma, address); +mem_cgroup_oom_enable (); ret =handle_pte_fault (&vmf);mem_cgroup_oom_disable ();Your patch is fixing the race but I'm wondering about the cost of these tlb flushes. Here we are flushing on a per page basis (architecture like
x86_64are smarter and flush more pages) but there is a request to flush a range of tlb entries each time a cow page is newly touched. I think there could be some bad impact here.Quoted 8 lines
Quoted 1 line
Another option would be to flush the range in
copy_pte_range()before unlocking the page table lock. This will flush entriesflush_tlb_mm()would later handle indup_mmap()but that will be called once per fork per cow VMA.But wouldn't this cause an unnecessary impact if most of the COW pages remain untouched (which I assume would be the usual case) and thus do not create a fault ?
Quoted 2 lines
I tried the attached patch which seems to fix the issue on
x86_64. Could you please give it a try on arm64 ?Your patch works fine on arm64 with a minor change. Thanks Laurent.
Hi, Vinayak and Laurent
I think the below change will impact the performance significantly. Becuase most of process has many vmas with cow flags. Flush the tlb in advance is not the better way to avoid the issue and it will call the
flush_tlb_mmlater.Quoted 8 lines
I think we can try the following way to do.
vm_write_begin(vma)copy_pte_rangevm_write_end(vma)The speculative page fault will return to grap the
mmap_semto run the nromal path. Any thought?Hi Zhong,
I agree that flushing the TLB could have a bad impact on the performance, but tagging the VMA when
copy_pte_range()is not fixing the issue as the VMA must be flagged until the PTE are flushed.Here is what happens:
CPU A CPU B CPU C
fork()copy_pte_range()
set PTE rdonly
got to next VMA.... PTE is seen rdonly PTE still writable . thread is writing to page . -> page fault. copy the page Thread writes to page
. . -> no page fault
. update the PTE
. flush TLB for that PTE
flush TLB PTE are now rdonly
Quoted 3 lines
Quoted 2 lines
So the write done by the CPU C is interfering with the page copy operation done by CPU B, leading to the data corruption.
I want to know the case if the CPU B has finished in front of the CPU C that the data still is vaild ?
If the CPU B has done the flush TLB then the CPU C will write data to the right page. If the CPU B has not yet done the flush of the TLB as the time the CPU C is writing data, then this roughly the same issue.
Anyway this is fixed with the patch I'm about to sent for testing on arm64.
Cheers,
Laurent.
Quoted 21 lines
This is to say, the
old_pagewill be changed from other cpu because of the access from other cpu.Maybe this is a stupid qestion :-)
Thanks,
zhong jiang.Quoted 12 lines
Flushing the PTE in
copy_pte_range()is fixing the issue as the CPU C is seeing the PTE as rdonly earlier. But this impacts performance.Another option, I'll work on is to flag
_allthe COWeligible_VMA before starting copying them and until the PTE are flushed on the CPU A. This way when the CPU B will page fault the speculative handler will abort because the VMA is in the way to be touched.But I need to ensure that all the calls to
copy_pte_range()are handling this correctly.Laurent.
.
Le 17/01/2019 à 16:51, zhong jiang a écrit :
Quoted 79 lines · Vinayak Menon
Quoted 64 lines · Laurent Dufour
Quoted 63 lines · Vinayak Menon
Quoted 24 lines · Vinayak Menon
Quoted 3 lines
Hi Laurent,
We are observing an issue with speculative page fault with the following test code on ARM64 (4.14 kernel, 8 cores).
With the patch below, we don't hit the issue.
From: Vinayak Menon <[email protected]>
Date: Mon, 14 Jan 2019 16:06:34 +0530
Subject: [PATCH] mm: flush stale tlb entries on speculative write faultIt is observed that the following scenario results in threads A and B of process 1 blocking on
pthread_mutex_lockforever after few iterations.CPU 1 CPU 2 CPU 3 Process 1, Process 1, Process 1, Thread A Thread B Thread Cwhile (1) { while (1) { while(1) { pthread_mutex_lock(l) pthread_mutex_lock(l) fork pthread_mutex_unlock(l) pthread_mutex_unlock(l) }} }
Quoted 9 lines
When from thread C,
copy_one_ptewrite-protects the parent pte (of lock l), stale tlb entries can exist with write permissions on one of the CPUs at least. This can create a problem if one of the threads A or B hits the write fault. Thoughdup_mmapcallsflush_tlb_mmaftercopy_page_range, since speculative page fault does not takemmap_semit can proceed further fixing a fault soon after CPU 3 doesptep_set_wrprotect. But the CPU with stale tlb entry can still modifyold_pageeven after it is copied tonew_pagebywp_page_copy, thus causing a corruption.Nice catch and thanks for your investigation!
There is a real synchronization issue here between
copy_page_range()and the speculative page fault handler. I didn't get it on PowerVM since the TLB are flushed whenarch_exit_lazy_mode()is called incopy_page_range()but now, I can get it when running onx86_64.Quoted 13 lines
Signed-off-by Vinayak Menon <[email protected]>--- mm/memory.c | 7 +++++++ 1 file changed, 7 insertions(+)mm/memory.c +0
-0@@ -4507,6 +4507,13 @@ int __handle_speculative_fault(struct mm_struct *mm, unsigned long address,return VM_FAULT_RETRY ; }mm/memory.c int __handle_speculative_fault(struct mm_struct *mm, unsigned long address, · line 4519
Quoted 10 lines
mm/memory.c +7
-0+/* +* Discard tlb entries created before ptep_set_wrprotect +* in copy_one_pte +*/ +if (flags &FAULT_FLAG_WRITE && !pte_write (vmf.orig_pte)) +flush_tlb_page (vmf.vma, address); +mem_cgroup_oom_enable (); ret =handle_pte_fault (&vmf);mem_cgroup_oom_disable ();Your patch is fixing the race but I'm wondering about the cost of these tlb flushes. Here we are flushing on a per page basis (architecture like
x86_64are smarter and flush more pages) but there is a request to flush a range of tlb entries each time a cow page is newly touched. I think there could be some bad impact here.Quoted 8 lines
Quoted 1 line
Another option would be to flush the range in
copy_pte_range()before unlocking the page table lock. This will flush entriesflush_tlb_mm()would later handle indup_mmap()but that will be called once per fork per cow VMA.But wouldn't this cause an unnecessary impact if most of the COW pages remain untouched (which I assume would be the usual case) and thus do not create a fault ?
Quoted 2 lines
I tried the attached patch which seems to fix the issue on
x86_64. Could you please give it a try on arm64 ?Your patch works fine on arm64 with a minor change. Thanks Laurent.
Hi, Vinayak and Laurent
I think the below change will impact the performance significantly. Becuase most of process has many vmas with cow flags. Flush the tlb in advance is not the better way to avoid the issue and it will call the
flush_tlb_mmlater.
Quoted 8 lines
I think we can try the following way to do.
vm_write_begin(vma)copy_pte_rangevm_write_end(vma)The speculative page fault will return to grap the
mmap_semto run the nromal path. Any thought?
Here is a new version of the patch fixing this issue. There is no additional TLB flush, all the fix is belonging on vm_write_{begin,end} calls.
I did some test on x86_64 and PowerPC but that needs to be double check on arm64.
Vinayak, Zhong, could you please give it a try ?
Thanks,
Laurent.
From 3be977febb9ff93d516a2d222cca4b5a52472a9f Mon Sep 17 00:00:00 2001
From: Laurent Dufour <[email protected]>
Date: Fri, 18 Jan 2019 16:19:08 +0100
Subject: [PATCH] mm: protect against PTE changes done by dup_mmap()
Vinayak Menon and Ganesh Mahendran reported that the following scenario may lead to thread being blocked due to data corruption:
CPU 1 CPU 2 CPU 3
Process 1, Process 1, Process 1,
Thread A Thread B Thread C while (1) { while (1) { while(1) {
pthread_mutex_lock(l) pthread_mutex_lock(l) fork
pthread_mutex_unlock(l) pthread_mutex_unlock(l) }
} }In the details this happens because :
CPU 1 CPU 2 CPU 3
fork()
copy_pte_range()
set PTE rdonly
got to next VMA...
. PTE is seen rdonly PTE still writable
. thread is writing to page
. -> page fault
. copy the page Thread writes to page
. . -> no page fault
. update the PTE
. flush TLB for that PTE
flush TLB PTE are now rdonly
So the write done by the CPU 3 is interfering with the page copy operation done by CPU 2, leading to the data corruption.
To avoid this we mark all the VMA involved in the COW mechanism as changing by calling vm_write_begin(). This ensures that the speculative page fault handler will not try to handle a fault on these pages.
The marker is set until the TLB is flushed, ensuring that all the CPUs will now see the PTE as not writable.
Once the TLB is flush, the marker is removed by calling vm_write_end().
The variable last is used to keep tracked of the latest VMA marked to handle the error path where part of the VMA may have been marked.
--- kernel/fork.c | 30 ++++++++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-)
kernel/fork.c +28 -2
@@ -395,7 +395,7 @@ EXPORT_SYMBOL(free_task); static __latent_entropy int dup_mmap(struct mm_struct *mm,struct mm_struct *oldmm) { -struct vm_area_struct *mpnt, *tmp, *prev, **pprev; +struct vm_area_struct *mpnt, *tmp, *prev, **pprev, *last =NULL ;struct rb_node **rb_link, *rb_parent;int retval;unsigned long charge;
@@ -515,8 +515,18 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, rb_parent = &tmp->vm_rb; mm->map_count++; -if (!(tmp->vm_flags &VM_WIPEONFORK )) +if (!(tmp->vm_flags &VM_WIPEONFORK )) { +if (IS_ENABLED (CONFIG_SPECULATIVE_PAGE_FAULT )) { +/* +* Mark this VMA as changing to prevent the +* speculative page fault hanlder to process +* it until the TLB are flushed below. +*/ + last = mpnt; +vm_write_begin (mpnt); + } retval =copy_page_range (mm, oldmm, mpnt); + }if (tmp->vm_ops && tmp->vm_ops->open) tmp->vm_ops->open (tmp);
@@ -530,6 +540,22 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, out:up_write (&mm->mmap_sem);flush_tlb_mm (oldmm); + +if (IS_ENABLED (CONFIG_SPECULATIVE_PAGE_FAULT )) { +/* +* Since the TLB has been flush, we can safely unmark the +* copied VMAs and allows the speculative page fault handler to +* process them again. +* Walk back the VMA list from the last marked VMA. +*/ +for (; last; last = last->vm_prev) { +if (last->vm_flags &VM_DONTCOPY ) +continue ; +if (!(last->vm_flags &VM_WIPEONFORK )) +vm_write_end (last); + } + } +up_write (&oldmm->mmap_sem);dup_userfaultfd_complete (&uf); fail_uprobe_end:
Signature · 1 line
2.20.1
Quoted 86 lines · Laurent Dufour
Quoted 85 lines · zhong jiang
Quoted 64 lines · Vinayak Menon
Quoted 63 lines · Laurent Dufour
Quoted 24 lines · Vinayak Menon
Quoted 3 lines · Vinayak Menon
Hi Laurent,
We are observing an issue with speculative page fault with the following test code on ARM64 (4.14 kernel, 8 cores).
With the patch below, we don't hit the issue.
From: Vinayak Menon <[email protected]>
Date: Mon, 14 Jan 2019 16:06:34 +0530
Subject: [PATCH] mm: flush stale tlb entries on speculative write faultIt is observed that the following scenario results in threads A and B of process 1 blocking on
pthread_mutex_lockforever after few iterations.CPU 1 CPU 2 CPU 3 Process 1, Process 1, Process 1, Thread A Thread B Thread Cwhile (1) { while (1) { while(1) { pthread_mutex_lock(l) pthread_mutex_lock(l) fork pthread_mutex_unlock(l) pthread_mutex_unlock(l) }} }
Quoted 9 lines
When from thread C,
copy_one_ptewrite-protects the parent pte (of lock l), stale tlb entries can exist with write permissions on one of the CPUs at least. This can create a problem if one of the threads A or B hits the write fault. Thoughdup_mmapcallsflush_tlb_mmaftercopy_page_range, since speculative page fault does not takemmap_semit can proceed further fixing a fault soon after CPU 3 doesptep_set_wrprotect. But the CPU with stale tlb entry can still modifyold_pageeven after it is copied tonew_pagebywp_page_copy, thus causing a corruption.Nice catch and thanks for your investigation!
There is a real synchronization issue here between
copy_page_range()and the speculative page fault handler. I didn't get it on PowerVM since the TLB are flushed whenarch_exit_lazy_mode()is called incopy_page_range()but now, I can get it when running onx86_64.Quoted 13 lines
Signed-off-by Vinayak Menon <[email protected]>--- mm/memory.c | 7 +++++++ 1 file changed, 7 insertions(+)mm/memory.c +0
-0@@ -4507,6 +4507,13 @@ int __handle_speculative_fault(struct mm_struct *mm, unsigned long address,return VM_FAULT_RETRY ; }mm/memory.c int __handle_speculative_fault(struct mm_struct *mm, unsigned long address, · line 4519
Quoted 10 lines
mm/memory.c +7
-0+/* +* Discard tlb entries created before ptep_set_wrprotect +* in copy_one_pte +*/ +if (flags &FAULT_FLAG_WRITE && !pte_write (vmf.orig_pte)) +flush_tlb_page (vmf.vma, address); +mem_cgroup_oom_enable (); ret =handle_pte_fault (&vmf);mem_cgroup_oom_disable ();Your patch is fixing the race but I'm wondering about the cost of these tlb flushes. Here we are flushing on a per page basis (architecture like
x86_64are smarter and flush more pages) but there is a request to flush a range of tlb entries each time a cow page is newly touched. I think there could be some bad impact here.Quoted 8 lines
Quoted 1 line
Another option would be to flush the range in
copy_pte_range()before unlocking the page table lock. This will flush entriesflush_tlb_mm()would later handle indup_mmap()but that will be called once per fork per cow VMA.But wouldn't this cause an unnecessary impact if most of the COW pages remain untouched (which I assume would be the usual case) and thus do not create a fault ?
Quoted 2 lines
I tried the attached patch which seems to fix the issue on
x86_64. Could you please give it a try on arm64 ?Your patch works fine on arm64 with a minor change. Thanks Laurent.
Hi, Vinayak and Laurent
I think the below change will impact the performance significantly. Becuase most of process has many vmas with cow flags. Flush the tlb in advance is not the better way to avoid the issue and it will call the
flush_tlb_mmlater.I think we can try the following way to do.
vm_write_begin(vma)copy_pte_rangevm_write_end(vma)
Quoted 9 lines
Quoted 2 lines
The speculative page fault will return to grap the
mmap_semto run the nromal path. Any thought?Here is a new version of the patch fixing this issue. There is no additional TLB flush, all the fix is belonging on vm_write_{begin,end} calls.
I did some test on
x86_64and PowerPC but that needs to be double check on arm64.Vinayak, Zhong, could you please give it a try ?
Thanks, look good to me. I will try it.
Sincerely,
zhong jiang
Quoted 3 lines
Thanks,
Laurent.
Quoted 86 lines · Laurent Dufour
Quoted 85 lines · zhong jiang
Quoted 64 lines · Vinayak Menon
Quoted 63 lines · Laurent Dufour
Quoted 24 lines · Vinayak Menon
Quoted 3 lines · Vinayak Menon
Hi Laurent,
We are observing an issue with speculative page fault with the following test code on ARM64 (4.14 kernel, 8 cores).
With the patch below, we don't hit the issue.
From: Vinayak Menon <[email protected]>
Date: Mon, 14 Jan 2019 16:06:34 +0530
Subject: [PATCH] mm: flush stale tlb entries on speculative write faultIt is observed that the following scenario results in threads A and B of process 1 blocking on
pthread_mutex_lockforever after few iterations.CPU 1 CPU 2 CPU 3 Process 1, Process 1, Process 1, Thread A Thread B Thread Cwhile (1) { while (1) { while(1) { pthread_mutex_lock(l) pthread_mutex_lock(l) fork pthread_mutex_unlock(l) pthread_mutex_unlock(l) }} }
Quoted 9 lines
When from thread C,
copy_one_ptewrite-protects the parent pte (of lock l), stale tlb entries can exist with write permissions on one of the CPUs at least. This can create a problem if one of the threads A or B hits the write fault. Thoughdup_mmapcallsflush_tlb_mmaftercopy_page_range, since speculative page fault does not takemmap_semit can proceed further fixing a fault soon after CPU 3 doesptep_set_wrprotect. But the CPU with stale tlb entry can still modifyold_pageeven after it is copied tonew_pagebywp_page_copy, thus causing a corruption.Nice catch and thanks for your investigation!
There is a real synchronization issue here between
copy_page_range()and the speculative page fault handler. I didn't get it on PowerVM since the TLB are flushed whenarch_exit_lazy_mode()is called incopy_page_range()but now, I can get it when running onx86_64.Quoted 13 lines
Signed-off-by Vinayak Menon <[email protected]>--- mm/memory.c | 7 +++++++ 1 file changed, 7 insertions(+)mm/memory.c +0
-0@@ -4507,6 +4507,13 @@ int __handle_speculative_fault(struct mm_struct *mm, unsigned long address,return VM_FAULT_RETRY ; }mm/memory.c int __handle_speculative_fault(struct mm_struct *mm, unsigned long address, · line 4519
Quoted 10 lines
mm/memory.c +7
-0+/* +* Discard tlb entries created before ptep_set_wrprotect +* in copy_one_pte +*/ +if (flags &FAULT_FLAG_WRITE && !pte_write (vmf.orig_pte)) +flush_tlb_page (vmf.vma, address); +mem_cgroup_oom_enable (); ret =handle_pte_fault (&vmf);mem_cgroup_oom_disable ();Your patch is fixing the race but I'm wondering about the cost of these tlb flushes. Here we are flushing on a per page basis (architecture like
x86_64are smarter and flush more pages) but there is a request to flush a range of tlb entries each time a cow page is newly touched. I think there could be some bad impact here.Quoted 8 lines
Quoted 1 line
Another option would be to flush the range in
copy_pte_range()before unlocking the page table lock. This will flush entriesflush_tlb_mm()would later handle indup_mmap()but that will be called once per fork per cow VMA.But wouldn't this cause an unnecessary impact if most of the COW pages remain untouched (which I assume would be the usual case) and thus do not create a fault ?
Quoted 2 lines
I tried the attached patch which seems to fix the issue on
x86_64. Could you please give it a try on arm64 ?Your patch works fine on arm64 with a minor change. Thanks Laurent.
Hi, Vinayak and Laurent
I think the below change will impact the performance significantly. Becuase most of process has many vmas with cow flags. Flush the tlb in advance is not the better way to avoid the issue and it will call the
flush_tlb_mmlater.I think we can try the following way to do.
vm_write_begin(vma)copy_pte_rangevm_write_end(vma)
Quoted 9 lines
Quoted 2 lines
The speculative page fault will return to grap the
mmap_semto run the nromal path. Any thought?Here is a new version of the patch fixing this issue. There is no additional TLB flush, all the fix is belonging on vm_write_{begin,end} calls.
I did some test on
x86_64and PowerPC but that needs to be double check on arm64.Vinayak, Zhong, could you please give it a try ?
Hi Laurent
I apply the patch you had attached and none of any abnormal thing came in two days. It is feasible to fix the issue.
but It will better to filter the condition by is_cow_mapping. is it right?
for example:
if (is_cow_mapping(mnpt->vm_flags)) {
........
}Thanks,
zhong jiang
Quoted 3 lines
Thanks,
Laurent.
Le 22/01/2019 à 17:22, zhong jiang a écrit :
Quoted 92 lines · Laurent Dufour
Quoted 91 lines · zhong jiang
Quoted 79 lines · Vinayak Menon
Quoted 64 lines · Laurent Dufour
Quoted 63 lines · Vinayak Menon
Quoted 24 lines · Vinayak Menon
Quoted 3 lines
Hi Laurent,
We are observing an issue with speculative page fault with the following test code on ARM64 (4.14 kernel, 8 cores).
With the patch below, we don't hit the issue.
From: Vinayak Menon <[email protected]>
Date: Mon, 14 Jan 2019 16:06:34 +0530
Subject: [PATCH] mm: flush stale tlb entries on speculative write faultIt is observed that the following scenario results in threads A and B of process 1 blocking on
pthread_mutex_lockforever after few iterations.CPU 1 CPU 2 CPU 3 Process 1, Process 1, Process 1, Thread A Thread B Thread Cwhile (1) { while (1) { while(1) { pthread_mutex_lock(l) pthread_mutex_lock(l) fork pthread_mutex_unlock(l) pthread_mutex_unlock(l) }} }
Quoted 9 lines
When from thread C,
copy_one_ptewrite-protects the parent pte (of lock l), stale tlb entries can exist with write permissions on one of the CPUs at least. This can create a problem if one of the threads A or B hits the write fault. Thoughdup_mmapcallsflush_tlb_mmaftercopy_page_range, since speculative page fault does not takemmap_semit can proceed further fixing a fault soon after CPU 3 doesptep_set_wrprotect. But the CPU with stale tlb entry can still modifyold_pageeven after it is copied tonew_pagebywp_page_copy, thus causing a corruption.Nice catch and thanks for your investigation!
There is a real synchronization issue here between
copy_page_range()and the speculative page fault handler. I didn't get it on PowerVM since the TLB are flushed whenarch_exit_lazy_mode()is called incopy_page_range()but now, I can get it when running onx86_64.Quoted 13 lines
Signed-off-by Vinayak Menon <[email protected]>--- mm/memory.c | 7 +++++++ 1 file changed, 7 insertions(+)mm/memory.c +0
-0@@ -4507,6 +4507,13 @@ int __handle_speculative_fault(struct mm_struct *mm, unsigned long address,return VM_FAULT_RETRY ; }mm/memory.c int __handle_speculative_fault(struct mm_struct *mm, unsigned long address, · line 4519
Quoted 10 lines
mm/memory.c +7
-0+/* +* Discard tlb entries created before ptep_set_wrprotect +* in copy_one_pte +*/ +if (flags &FAULT_FLAG_WRITE && !pte_write (vmf.orig_pte)) +flush_tlb_page (vmf.vma, address); +mem_cgroup_oom_enable (); ret =handle_pte_fault (&vmf);mem_cgroup_oom_disable ();Your patch is fixing the race but I'm wondering about the cost of these tlb flushes. Here we are flushing on a per page basis (architecture like
x86_64are smarter and flush more pages) but there is a request to flush a range of tlb entries each time a cow page is newly touched. I think there could be some bad impact here.Quoted 8 lines
Quoted 1 line
Another option would be to flush the range in
copy_pte_range()before unlocking the page table lock. This will flush entriesflush_tlb_mm()would later handle indup_mmap()but that will be called once per fork per cow VMA.But wouldn't this cause an unnecessary impact if most of the COW pages remain untouched (which I assume would be the usual case) and thus do not create a fault ?
Quoted 2 lines
I tried the attached patch which seems to fix the issue on
x86_64. Could you please give it a try on arm64 ?Your patch works fine on arm64 with a minor change. Thanks Laurent.
Hi, Vinayak and Laurent
I think the below change will impact the performance significantly. Becuase most of process has many vmas with cow flags. Flush the tlb in advance is not the better way to avoid the issue and it will call the
flush_tlb_mmlater.Quoted 8 lines
I think we can try the following way to do.
vm_write_begin(vma)copy_pte_rangevm_write_end(vma)The speculative page fault will return to grap the
mmap_semto run the nromal path. Any thought?Here is a new version of the patch fixing this issue. There is no additional TLB flush, all the fix is belonging on vm_write_{begin,end} calls.
Quoted 7 lines
Quoted 4 lines
I did some test on
x86_64and PowerPC but that needs to be double check on arm64.Vinayak, Zhong, could you please give it a try ?
Hi Laurent
I apply the patch you had attached and none of any abnormal thing came in two days. It is feasible to fix the issue.
Good news !
Quoted 9 lines
but It will better to filter the condition by
is_cow_mapping. is it right?for example:
if (is_cow_mapping(mnpt->vm_flags)) { ........ }
That's doable for sure but I don't think this has to be introduce in dup_mmap().
Unless there is a real performance benefit to do so, I don't think dup_mmap() has to mimic underlying checks done in copy_page_range().
Cheers,
Laurent.
Quoted 106 lines · Laurent Dufour
Quoted 92 lines · zhong jiang
Quoted 91 lines · Laurent Dufour
Quoted 79 lines · zhong jiang
Quoted 64 lines · Vinayak Menon
Quoted 63 lines · Laurent Dufour
Quoted 24 lines · Vinayak Menon
Quoted 3 lines · Vinayak Menon
Hi Laurent,
We are observing an issue with speculative page fault with the following test code on ARM64 (4.14 kernel, 8 cores).
With the patch below, we don't hit the issue.
From: Vinayak Menon <[email protected]>
Date: Mon, 14 Jan 2019 16:06:34 +0530
Subject: [PATCH] mm: flush stale tlb entries on speculative write faultIt is observed that the following scenario results in threads A and B of process 1 blocking on
pthread_mutex_lockforever after few iterations.CPU 1 CPU 2 CPU 3 Process 1, Process 1, Process 1, Thread A Thread B Thread Cwhile (1) { while (1) { while(1) { pthread_mutex_lock(l) pthread_mutex_lock(l) fork pthread_mutex_unlock(l) pthread_mutex_unlock(l) }} }
Quoted 9 lines
When from thread C,
copy_one_ptewrite-protects the parent pte (of lock l), stale tlb entries can exist with write permissions on one of the CPUs at least. This can create a problem if one of the threads A or B hits the write fault. Thoughdup_mmapcallsflush_tlb_mmaftercopy_page_range, since speculative page fault does not takemmap_semit can proceed further fixing a fault soon after CPU 3 doesptep_set_wrprotect. But the CPU with stale tlb entry can still modifyold_pageeven after it is copied tonew_pagebywp_page_copy, thus causing a corruption.Nice catch and thanks for your investigation!
There is a real synchronization issue here between
copy_page_range()and the speculative page fault handler. I didn't get it on PowerVM since the TLB are flushed whenarch_exit_lazy_mode()is called incopy_page_range()but now, I can get it when running onx86_64.Quoted 13 lines
Signed-off-by Vinayak Menon <[email protected]>--- mm/memory.c | 7 +++++++ 1 file changed, 7 insertions(+)mm/memory.c +0
-0@@ -4507,6 +4507,13 @@ int __handle_speculative_fault(struct mm_struct *mm, unsigned long address,return VM_FAULT_RETRY ; }mm/memory.c int __handle_speculative_fault(struct mm_struct *mm, unsigned long address, · line 4519
Quoted 10 lines
mm/memory.c +7
-0+/* +* Discard tlb entries created before ptep_set_wrprotect +* in copy_one_pte +*/ +if (flags &FAULT_FLAG_WRITE && !pte_write (vmf.orig_pte)) +flush_tlb_page (vmf.vma, address); +mem_cgroup_oom_enable (); ret =handle_pte_fault (&vmf);mem_cgroup_oom_disable ();Your patch is fixing the race but I'm wondering about the cost of these tlb flushes. Here we are flushing on a per page basis (architecture like
x86_64are smarter and flush more pages) but there is a request to flush a range of tlb entries each time a cow page is newly touched. I think there could be some bad impact here.Quoted 8 lines
Quoted 1 line
Another option would be to flush the range in
copy_pte_range()before unlocking the page table lock. This will flush entriesflush_tlb_mm()would later handle indup_mmap()but that will be called once per fork per cow VMA.But wouldn't this cause an unnecessary impact if most of the COW pages remain untouched (which I assume would be the usual case) and thus do not create a fault ?
Quoted 2 lines
I tried the attached patch which seems to fix the issue on
x86_64. Could you please give it a try on arm64 ?Your patch works fine on arm64 with a minor change. Thanks Laurent.
Hi, Vinayak and Laurent
I think the below change will impact the performance significantly. Becuase most of process has many vmas with cow flags. Flush the tlb in advance is not the better way to avoid the issue and it will call the
flush_tlb_mmlater.Quoted 8 lines
I think we can try the following way to do.
vm_write_begin(vma)copy_pte_rangevm_write_end(vma)The speculative page fault will return to grap the
mmap_semto run the nromal path. Any thought?Here is a new version of the patch fixing this issue. There is no additional TLB flush, all the fix is belonging on vm_write_{begin,end} calls.
Quoted 7 lines
Quoted 4 lines
I did some test on
x86_64and PowerPC but that needs to be double check on arm64.Vinayak, Zhong, could you please give it a try ?
Hi Laurent
I apply the patch you had attached and none of any abnormal thing came in two days. It is feasible to fix the issue.
Good news !
Quoted 3 lines
but It will better to filter the condition by
is_cow_mapping. is it right?
Quoted 9 lines
Quoted 5 lines
for example:
if (is_cow_mapping(mnpt->vm_flags)) { ........ }That's doable for sure but I don't think this has to be introduce in
dup_mmap().
Unless there is a real performance benefit to do so, I don't thinkdup_mmap()has to mimic underlying checks done incopy_page_range().
Hi, Laurent
I test the performace with microbench after appling the patch. I find the page fault latency will increase about 8% than before. I think we should use is_cow_mapping to waken the impact and I will try it out.
or we can use the following solution to replace as Vinayak has said.
if (flags & FAULT_FLAG_WRITE && !pte_write(vmf.orig_pte))
return VM_FAULT_RETRY;
Even though it will influence the performance of SPF, but at least it does not bring in any negative impact. Any thought?
Thanks,
Quoted 6 lines
Cheers,
Laurent..
Le 25/01/2019 à 13:32, zhong jiang a écrit :
Quoted 117 lines · Laurent Dufour
Quoted 106 lines · zhong jiang
Quoted 92 lines · Laurent Dufour
Quoted 91 lines · zhong jiang
Quoted 79 lines · Vinayak Menon
Quoted 64 lines · Laurent Dufour
Quoted 63 lines · Vinayak Menon
Quoted 24 lines · Vinayak Menon
Quoted 3 lines
Hi Laurent,
We are observing an issue with speculative page fault with the following test code on ARM64 (4.14 kernel, 8 cores).
With the patch below, we don't hit the issue.
From: Vinayak Menon <[email protected]>
Date: Mon, 14 Jan 2019 16:06:34 +0530
Subject: [PATCH] mm: flush stale tlb entries on speculative write faultIt is observed that the following scenario results in threads A and B of process 1 blocking on
pthread_mutex_lockforever after few iterations.CPU 1 CPU 2 CPU 3 Process 1, Process 1, Process 1, Thread A Thread B Thread Cwhile (1) { while (1) { while(1) { pthread_mutex_lock(l) pthread_mutex_lock(l) fork pthread_mutex_unlock(l) pthread_mutex_unlock(l) }} }
Quoted 9 lines
When from thread C,
copy_one_ptewrite-protects the parent pte (of lock l), stale tlb entries can exist with write permissions on one of the CPUs at least. This can create a problem if one of the threads A or B hits the write fault. Thoughdup_mmapcallsflush_tlb_mmaftercopy_page_range, since speculative page fault does not takemmap_semit can proceed further fixing a fault soon after CPU 3 doesptep_set_wrprotect. But the CPU with stale tlb entry can still modifyold_pageeven after it is copied tonew_pagebywp_page_copy, thus causing a corruption.Nice catch and thanks for your investigation!
There is a real synchronization issue here between
copy_page_range()and the speculative page fault handler. I didn't get it on PowerVM since the TLB are flushed whenarch_exit_lazy_mode()is called incopy_page_range()but now, I can get it when running onx86_64.Quoted 13 lines
Signed-off-by Vinayak Menon <[email protected]>--- mm/memory.c | 7 +++++++ 1 file changed, 7 insertions(+)mm/memory.c +0
-0@@ -4507,6 +4507,13 @@ int __handle_speculative_fault(struct mm_struct *mm, unsigned long address,return VM_FAULT_RETRY ; }mm/memory.c int __handle_speculative_fault(struct mm_struct *mm, unsigned long address, · line 4519
Quoted 10 lines
mm/memory.c +7
-0+/* +* Discard tlb entries created before ptep_set_wrprotect +* in copy_one_pte +*/ +if (flags &FAULT_FLAG_WRITE && !pte_write (vmf.orig_pte)) +flush_tlb_page (vmf.vma, address); +mem_cgroup_oom_enable (); ret =handle_pte_fault (&vmf);mem_cgroup_oom_disable ();Your patch is fixing the race but I'm wondering about the cost of these tlb flushes. Here we are flushing on a per page basis (architecture like
x86_64are smarter and flush more pages) but there is a request to flush a range of tlb entries each time a cow page is newly touched. I think there could be some bad impact here.Quoted 8 lines
Quoted 1 line
Another option would be to flush the range in
copy_pte_range()before unlocking the page table lock. This will flush entriesflush_tlb_mm()would later handle indup_mmap()but that will be called once per fork per cow VMA.But wouldn't this cause an unnecessary impact if most of the COW pages remain untouched (which I assume would be the usual case) and thus do not create a fault ?
Quoted 2 lines
I tried the attached patch which seems to fix the issue on
x86_64. Could you please give it a try on arm64 ?Your patch works fine on arm64 with a minor change. Thanks Laurent.
Hi, Vinayak and Laurent
I think the below change will impact the performance significantly. Becuase most of process has many vmas with cow flags. Flush the tlb in advance is not the better way to avoid the issue and it will call the
flush_tlb_mmlater.Quoted 8 lines
I think we can try the following way to do.
vm_write_begin(vma)copy_pte_rangevm_write_end(vma)The speculative page fault will return to grap the
mmap_semto run the nromal path. Any thought?Here is a new version of the patch fixing this issue. There is no additional TLB flush, all the fix is belonging on vm_write_{begin,end} calls.
Quoted 7 lines
Quoted 4 lines
I did some test on
x86_64and PowerPC but that needs to be double check on arm64.Vinayak, Zhong, could you please give it a try ?
Hi Laurent
I apply the patch you had attached and none of any abnormal thing came in two days. It is feasible to fix the issue.
Good news !
Quoted 3 lines
but It will better to filter the condition by
is_cow_mapping. is it right?Quoted 9 lines
Quoted 5 lines
for example:
if (is_cow_mapping(mnpt->vm_flags)) { ........ }That's doable for sure but I don't think this has to be introduce in
dup_mmap().
Unless there is a real performance benefit to do so, I don't thinkdup_mmap()has to mimic underlying checks done incopy_page_range().
Quoted 5 lines
Hi, Laurent
I test the performace with microbench after appling the patch. I find the page fault latency will increase about 8% than before. I think we should use
is_cow_mappingto waken the impact and I will try it out.
That's interesting, I would not expect such a higher latency assuming that most of the area not in copied on write are also not managed by the speculative page fault handler (file mapping, etc.). Anyway I'm looking forward to see the result with additional is_cow_mapping() check.
Quoted 7 lines
or we can use the following solution to replace as Vinayak has said.
if (flags &
FAULT_FLAG_WRITE&& !pte_write(vmf.orig_pte))return VM_FAULT_RETRY;Even though it will influence the performance of SPF, but at least it does not bring in any negative impact. Any thought?
I don't agree, this checks will completely by pass the SPF handler for all the COW areas, even if there is no race situation.
Cheers,
Laurent.
Quoted 12 lines
Thanks,
Quoted 6 lines
Cheers,
Laurent..
Quoted 118 lines · Laurent Dufour
Quoted 117 lines · zhong jiang
Quoted 115 lines · Laurent Dufour
Quoted 92 lines · zhong jiang
Quoted 91 lines · Laurent Dufour
Quoted 79 lines · zhong jiang
Quoted 64 lines · Vinayak Menon
Quoted 63 lines · Laurent Dufour
Quoted 24 lines · Vinayak Menon
Quoted 3 lines · Vinayak Menon
Hi Laurent,
We are observing an issue with speculative page fault with the following test code on ARM64 (4.14 kernel, 8 cores).
With the patch below, we don't hit the issue.
From: Vinayak Menon <[email protected]>
Date: Mon, 14 Jan 2019 16:06:34 +0530
Subject: [PATCH] mm: flush stale tlb entries on speculative write faultIt is observed that the following scenario results in threads A and B of process 1 blocking on
pthread_mutex_lockforever after few iterations.CPU 1 CPU 2 CPU 3 Process 1, Process 1, Process 1, Thread A Thread B Thread Cwhile (1) { while (1) { while(1) { pthread_mutex_lock(l) pthread_mutex_lock(l) fork pthread_mutex_unlock(l) pthread_mutex_unlock(l) }} }
Quoted 9 lines
When from thread C,
copy_one_ptewrite-protects the parent pte (of lock l), stale tlb entries can exist with write permissions on one of the CPUs at least. This can create a problem if one of the threads A or B hits the write fault. Thoughdup_mmapcallsflush_tlb_mmaftercopy_page_range, since speculative page fault does not takemmap_semit can proceed further fixing a fault soon after CPU 3 doesptep_set_wrprotect. But the CPU with stale tlb entry can still modifyold_pageeven after it is copied tonew_pagebywp_page_copy, thus causing a corruption.Nice catch and thanks for your investigation!
There is a real synchronization issue here between
copy_page_range()and the speculative page fault handler. I didn't get it on PowerVM since the TLB are flushed whenarch_exit_lazy_mode()is called incopy_page_range()but now, I can get it when running onx86_64.Quoted 13 lines
Signed-off-by Vinayak Menon <[email protected]>--- mm/memory.c | 7 +++++++ 1 file changed, 7 insertions(+)mm/memory.c +0
-0@@ -4507,6 +4507,13 @@ int __handle_speculative_fault(struct mm_struct *mm, unsigned long address,return VM_FAULT_RETRY ; }mm/memory.c int __handle_speculative_fault(struct mm_struct *mm, unsigned long address, · line 4519
Quoted 10 lines
mm/memory.c +7
-0+/* +* Discard tlb entries created before ptep_set_wrprotect +* in copy_one_pte +*/ +if (flags &FAULT_FLAG_WRITE && !pte_write (vmf.orig_pte)) +flush_tlb_page (vmf.vma, address); +mem_cgroup_oom_enable (); ret =handle_pte_fault (&vmf);mem_cgroup_oom_disable ();Your patch is fixing the race but I'm wondering about the cost of these tlb flushes. Here we are flushing on a per page basis (architecture like
x86_64are smarter and flush more pages) but there is a request to flush a range of tlb entries each time a cow page is newly touched. I think there could be some bad impact here.Quoted 8 lines
Quoted 1 line
Another option would be to flush the range in
copy_pte_range()before unlocking the page table lock. This will flush entriesflush_tlb_mm()would later handle indup_mmap()but that will be called once per fork per cow VMA.But wouldn't this cause an unnecessary impact if most of the COW pages remain untouched (which I assume would be the usual case) and thus do not create a fault ?
Quoted 2 lines
I tried the attached patch which seems to fix the issue on
x86_64. Could you please give it a try on arm64 ?Your patch works fine on arm64 with a minor change. Thanks Laurent.
Hi, Vinayak and Laurent
I think the below change will impact the performance significantly. Becuase most of process has many vmas with cow flags. Flush the tlb in advance is not the better way to avoid the issue and it will call the
flush_tlb_mmlater.Quoted 8 lines
I think we can try the following way to do.
vm_write_begin(vma)copy_pte_rangevm_write_end(vma)The speculative page fault will return to grap the
mmap_semto run the nromal path. Any thought?Here is a new version of the patch fixing this issue. There is no additional TLB flush, all the fix is belonging on vm_write_{begin,end} calls.
Quoted 7 lines
Quoted 4 lines
I did some test on
x86_64and PowerPC but that needs to be double check on arm64.Vinayak, Zhong, could you please give it a try ?
Hi Laurent
I apply the patch you had attached and none of any abnormal thing came in two days. It is feasible to fix the issue.
Good news !
Quoted 8 lines
but It will better to filter the condition by
is_cow_mapping. is it right?for example:
if (is_cow_mapping(mnpt->vm_flags)) { ........ }That's doable for sure but I don't think this has to be introduce in
dup_mmap().
Unless there is a real performance benefit to do so, I don't thinkdup_mmap()has to mimic underlying checks done incopy_page_range().
Quoted 8 lines
Quoted 5 lines
Hi, Laurent
I test the performace with microbench after appling the patch. I find the page fault latency will increase about 8% than before. I think we should use
is_cow_mappingto waken the impact and I will try it out.That's interesting, I would not expect such a higher latency assuming that most of the area not in copied on write are also not managed by the speculative page fault handler (file mapping, etc.). Anyway I'm looking forward to see the result with additional
is_cow_mapping()check.
I test the performance again. It is the protect error access latency in lat_sig.c that it will result in a drop of 8% in that testcase. The page fault latency, In fact, does not impact the performace. It seems to just the fluctuation.
Thanks,
zhong jiang
Quoted 27 lines
Quoted 7 lines
or we can use the following solution to replace as Vinayak has said.
if (flags &
FAULT_FLAG_WRITE&& !pte_write(vmf.orig_pte))return VM_FAULT_RETRY;Even though it will influence the performance of SPF, but at least it does not bring in any negative impact. Any thought?
I don't agree, this checks will completely by pass the SPF handler for all the COW areas, even if there is no race situation.
Cheers,
Laurent.Quoted 12 lines
Thanks,
Quoted 6 lines
Cheers,
Laurent..
Le 28/01/2019 à 15:09, zhong jiang a écrit :
Quoted 121 lines · Laurent Dufour
Quoted 120 lines · zhong jiang
Quoted 119 lines · Laurent Dufour
Quoted 106 lines · zhong jiang
Quoted 92 lines · Laurent Dufour
Quoted 91 lines · zhong jiang
Quoted 79 lines · Vinayak Menon
Quoted 64 lines · Laurent Dufour
Quoted 63 lines · Vinayak Menon
Quoted 24 lines · Vinayak Menon
Quoted 3 lines
Hi Laurent,
We are observing an issue with speculative page fault with the following test code on ARM64 (4.14 kernel, 8 cores).
With the patch below, we don't hit the issue.
From: Vinayak Menon <[email protected]>
Date: Mon, 14 Jan 2019 16:06:34 +0530
Subject: [PATCH] mm: flush stale tlb entries on speculative write faultIt is observed that the following scenario results in threads A and B of process 1 blocking on
pthread_mutex_lockforever after few iterations.CPU 1 CPU 2 CPU 3 Process 1, Process 1, Process 1, Thread A Thread B Thread Cwhile (1) { while (1) { while(1) { pthread_mutex_lock(l) pthread_mutex_lock(l) fork pthread_mutex_unlock(l) pthread_mutex_unlock(l) }} }
Quoted 9 lines
When from thread C,
copy_one_ptewrite-protects the parent pte (of lock l), stale tlb entries can exist with write permissions on one of the CPUs at least. This can create a problem if one of the threads A or B hits the write fault. Thoughdup_mmapcallsflush_tlb_mmaftercopy_page_range, since speculative page fault does not takemmap_semit can proceed further fixing a fault soon after CPU 3 doesptep_set_wrprotect. But the CPU with stale tlb entry can still modifyold_pageeven after it is copied tonew_pagebywp_page_copy, thus causing a corruption.Nice catch and thanks for your investigation!
There is a real synchronization issue here between
copy_page_range()and the speculative page fault handler. I didn't get it on PowerVM since the TLB are flushed whenarch_exit_lazy_mode()is called incopy_page_range()but now, I can get it when running onx86_64.Quoted 13 lines
Signed-off-by Vinayak Menon <[email protected]>--- mm/memory.c | 7 +++++++ 1 file changed, 7 insertions(+)mm/memory.c +0
-0@@ -4507,6 +4507,13 @@ int __handle_speculative_fault(struct mm_struct *mm, unsigned long address,return VM_FAULT_RETRY ; }mm/memory.c int __handle_speculative_fault(struct mm_struct *mm, unsigned long address, · line 4519
Quoted 10 lines
mm/memory.c +7
-0+/* +* Discard tlb entries created before ptep_set_wrprotect +* in copy_one_pte +*/ +if (flags &FAULT_FLAG_WRITE && !pte_write (vmf.orig_pte)) +flush_tlb_page (vmf.vma, address); +mem_cgroup_oom_enable (); ret =handle_pte_fault (&vmf);mem_cgroup_oom_disable ();Your patch is fixing the race but I'm wondering about the cost of these tlb flushes. Here we are flushing on a per page basis (architecture like
x86_64are smarter and flush more pages) but there is a request to flush a range of tlb entries each time a cow page is newly touched. I think there could be some bad impact here.Quoted 8 lines
Quoted 1 line
Another option would be to flush the range in
copy_pte_range()before unlocking the page table lock. This will flush entriesflush_tlb_mm()would later handle indup_mmap()but that will be called once per fork per cow VMA.But wouldn't this cause an unnecessary impact if most of the COW pages remain untouched (which I assume would be the usual case) and thus do not create a fault ?
Quoted 2 lines
I tried the attached patch which seems to fix the issue on
x86_64. Could you please give it a try on arm64 ?Your patch works fine on arm64 with a minor change. Thanks Laurent.
Hi, Vinayak and Laurent
I think the below change will impact the performance significantly. Becuase most of process has many vmas with cow flags. Flush the tlb in advance is not the better way to avoid the issue and it will call the
flush_tlb_mmlater.Quoted 8 lines
I think we can try the following way to do.
vm_write_begin(vma)copy_pte_rangevm_write_end(vma)The speculative page fault will return to grap the
mmap_semto run the nromal path. Any thought?Here is a new version of the patch fixing this issue. There is no additional TLB flush, all the fix is belonging on vm_write_{begin,end} calls.
Quoted 7 lines
Quoted 4 lines
I did some test on
x86_64and PowerPC but that needs to be double check on arm64.Vinayak, Zhong, could you please give it a try ?
Hi Laurent
I apply the patch you had attached and none of any abnormal thing came in two days. It is feasible to fix the issue.
Good news !
Quoted 3 lines
but It will better to filter the condition by
is_cow_mapping. is it right?Quoted 9 lines
Quoted 5 lines
for example:
if (is_cow_mapping(mnpt->vm_flags)) { ........ }That's doable for sure but I don't think this has to be introduce in
dup_mmap().
Unless there is a real performance benefit to do so, I don't thinkdup_mmap()has to mimic underlying checks done incopy_page_range().Hi, Laurent
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
I test the performace with microbench after appling the patch. I find the page fault latency will increase about 8% than before. I think we should use
is_cow_mappingto waken the impact and I will try it out.That's interesting, I would not expect such a higher latency assuming that most of the area not in copied on write are also not managed by the speculative page fault handler (file mapping, etc.). Anyway I'm looking forward to see the result with additional
is_cow_mapping()check.I test the performance again. It is the protect error access latency in
lat_sig.cthat it will result in a drop of 8% in that testcase.
What is that "protect error access latency in lat_sig.c" ?
Quoted 33 lines
The page fault latency, In fact, does not impact the performace. It seems to just the fluctuation.
Thanks,
zhong jiangQuoted 27 lines
Quoted 7 lines
or we can use the following solution to replace as Vinayak has said.
if (flags &
FAULT_FLAG_WRITE&& !pte_write(vmf.orig_pte))return VM_FAULT_RETRY;Even though it will influence the performance of SPF, but at least it does not bring in any negative impact. Any thought?
I don't agree, this checks will completely by pass the SPF handler for all the COW areas, even if there is no race situation.
Cheers,
Laurent.Quoted 12 lines
Thanks,
Quoted 6 lines
Cheers,
Laurent..
Quoted 126 lines · Laurent Dufour
Quoted 125 lines · zhong jiang
Quoted 124 lines · Laurent Dufour
Quoted 122 lines · zhong jiang
Quoted 106 lines · Laurent Dufour
Quoted 92 lines · zhong jiang
Quoted 91 lines · Laurent Dufour
Quoted 79 lines · zhong jiang
Quoted 64 lines · Vinayak Menon
Quoted 63 lines · Laurent Dufour
Quoted 24 lines · Vinayak Menon
Quoted 3 lines · Vinayak Menon
Hi Laurent,
We are observing an issue with speculative page fault with the following test code on ARM64 (4.14 kernel, 8 cores).
With the patch below, we don't hit the issue.
From: Vinayak Menon <[email protected]>
Date: Mon, 14 Jan 2019 16:06:34 +0530
Subject: [PATCH] mm: flush stale tlb entries on speculative write faultIt is observed that the following scenario results in threads A and B of process 1 blocking on
pthread_mutex_lockforever after few iterations.CPU 1 CPU 2 CPU 3 Process 1, Process 1, Process 1, Thread A Thread B Thread Cwhile (1) { while (1) { while(1) { pthread_mutex_lock(l) pthread_mutex_lock(l) fork pthread_mutex_unlock(l) pthread_mutex_unlock(l) }} }
Quoted 9 lines
When from thread C,
copy_one_ptewrite-protects the parent pte (of lock l), stale tlb entries can exist with write permissions on one of the CPUs at least. This can create a problem if one of the threads A or B hits the write fault. Thoughdup_mmapcallsflush_tlb_mmaftercopy_page_range, since speculative page fault does not takemmap_semit can proceed further fixing a fault soon after CPU 3 doesptep_set_wrprotect. But the CPU with stale tlb entry can still modifyold_pageeven after it is copied tonew_pagebywp_page_copy, thus causing a corruption.Nice catch and thanks for your investigation!
There is a real synchronization issue here between
copy_page_range()and the speculative page fault handler. I didn't get it on PowerVM since the TLB are flushed whenarch_exit_lazy_mode()is called incopy_page_range()but now, I can get it when running onx86_64.Quoted 13 lines
Signed-off-by Vinayak Menon <[email protected]>--- mm/memory.c | 7 +++++++ 1 file changed, 7 insertions(+)mm/memory.c +0
-0@@ -4507,6 +4507,13 @@ int __handle_speculative_fault(struct mm_struct *mm, unsigned long address,return VM_FAULT_RETRY ; }mm/memory.c int __handle_speculative_fault(struct mm_struct *mm, unsigned long address, · line 4519
Quoted 10 lines
mm/memory.c +7
-0+/* +* Discard tlb entries created before ptep_set_wrprotect +* in copy_one_pte +*/ +if (flags &FAULT_FLAG_WRITE && !pte_write (vmf.orig_pte)) +flush_tlb_page (vmf.vma, address); +mem_cgroup_oom_enable (); ret =handle_pte_fault (&vmf);mem_cgroup_oom_disable ();Your patch is fixing the race but I'm wondering about the cost of these tlb flushes. Here we are flushing on a per page basis (architecture like
x86_64are smarter and flush more pages) but there is a request to flush a range of tlb entries each time a cow page is newly touched. I think there could be some bad impact here.Quoted 8 lines
Quoted 1 line
Another option would be to flush the range in
copy_pte_range()before unlocking the page table lock. This will flush entriesflush_tlb_mm()would later handle indup_mmap()but that will be called once per fork per cow VMA.But wouldn't this cause an unnecessary impact if most of the COW pages remain untouched (which I assume would be the usual case) and thus do not create a fault ?
Quoted 2 lines
I tried the attached patch which seems to fix the issue on
x86_64. Could you please give it a try on arm64 ?Your patch works fine on arm64 with a minor change. Thanks Laurent.
Hi, Vinayak and Laurent
I think the below change will impact the performance significantly. Becuase most of process has many vmas with cow flags. Flush the tlb in advance is not the better way to avoid the issue and it will call the
flush_tlb_mmlater.Quoted 8 lines
I think we can try the following way to do.
vm_write_begin(vma)copy_pte_rangevm_write_end(vma)The speculative page fault will return to grap the
mmap_semto run the nromal path. Any thought?Here is a new version of the patch fixing this issue. There is no additional TLB flush, all the fix is belonging on vm_write_{begin,end} calls.
Quoted 7 lines
Quoted 4 lines
I did some test on
x86_64and PowerPC but that needs to be double check on arm64.Vinayak, Zhong, could you please give it a try ?
Hi Laurent
I apply the patch you had attached and none of any abnormal thing came in two days. It is feasible to fix the issue.
Good news !
Quoted 3 lines
but It will better to filter the condition by
is_cow_mapping. is it right?Quoted 9 lines
Quoted 5 lines
for example:
if (is_cow_mapping(mnpt->vm_flags)) { ........ }That's doable for sure but I don't think this has to be introduce in
dup_mmap().
Unless there is a real performance benefit to do so, I don't thinkdup_mmap()has to mimic underlying checks done incopy_page_range().Hi, Laurent
I test the performace with microbench after appling the patch. I find the page fault latency will increase about 8% than before. I think we should use
is_cow_mappingto waken the impact and I will try it out.
Quoted 6 lines
Quoted 3 lines
Quoted 2 lines
That's interesting, I would not expect such a higher latency assuming that most of the area not in copied on write are also not managed by the speculative page fault handler (file mapping, etc.). Anyway I'm looking forward to see the result with additional
is_cow_mapping()check.I test the performance again. It is the protect error access latency in
lat_sig.cthat it will result in a drop of 8% in that testcase.What is that "protect error access latency in
lat_sig.c" ?
It is the protect error access that the source code is lat_sig.c in microbench.
Quoted 37 lines
Quoted 33 lines
The page fault latency, In fact, does not impact the performace. It seems to just the fluctuation.
Thanks,
zhong jiangQuoted 27 lines
Quoted 7 lines
or we can use the following solution to replace as Vinayak has said.
if (flags &
FAULT_FLAG_WRITE&& !pte_write(vmf.orig_pte))return VM_FAULT_RETRY;Even though it will influence the performance of SPF, but at least it does not bring in any negative impact. Any thought?
I don't agree, this checks will completely by pass the SPF handler for all the COW areas, even if there is no race situation.
Cheers,
Laurent.Quoted 12 lines
Thanks,
Quoted 6 lines
Cheers,
Laurent..
.