mm: Respect FOLL_FORCE/FOLL_COW for thp
Subsystems (2)
PATCH 1/1 mm: Respect FOLL_FORCE/FOLL_COW for thp
In 19be0eaff ("mm: remove gup_flags FOLL_WRITE games from __get_user_pages()"), the mm code was changed from unsetting FOLL_WRITE after a COW was resolved to setting the (newly introduced) FOLL_COW instead. Simultaneously, the check in gup.c was updated to still allow writes with FOLL_FORCE set if FOLL_COW had also been set. However, a similar check in huge_memory.c was forgotten. As a result, remote memory writes to ro regions of memory backed by transparent huge pages cause an infinite loop in the kernel (handle_mm_fault sets FOLL_COW and returns 0 causing a retry, but follow_trans_huge_pmd bails out immidiately because (flags & FOLL_WRITE) && !pmd_write(*pmd) is true. While in this state, the process is stil SIGKILLable, but little else works (e.g. no ptrace attach, no other signals). This is easily reproduced with the following code (assuming thp are set to always):
```
#include <assert.h>
#include <fcntl.h>
#include <stdint.h>
#include <stdio.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <unistd.h>
#define TEST_SIZE 5 * 1024 * 1024
int main(void) {
int status;
pid_t child;
int fd = open("/proc/self/mem", O_RDWR);
void *addr =
mmap(NULL, TEST_SIZE, PROT_READ, MAP_ANONYMOUS | MAP_PRIVATE, 0, 0);
assert(addr != MAP_FAILED);
pid_t parent_pid = getpid();
if ((child = fork()) == 0) {
void *addr2 = mmap(NULL, TEST_SIZE, PROT_READ | PROT_WRITE,
MAP_ANONYMOUS | MAP_PRIVATE, 0, 0);
assert(addr2 != MAP_FAILED);
memset(addr2, 'a', TEST_SIZE);
pwrite(fd, addr2, TEST_SIZE, (uintptr_t)addr);
return 0;
}
assert(child == waitpid(child, &status, 0));
assert(WIFEXITED(status) && WEXITSTATUS(status) == 0);
return 0;
}
```Fix this by updating the instances in huge_memory.c analogously to the update in gup.c in the original commit. The same pattern existed in follow_devmap_pmd, so I have changed that location as well. However, I do not have a test case that for that code path.
--- mm/huge_memory.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
mm/huge_memory.c +12 -2
@@ -773,6 +773,16 @@ static void touch_pmd(struct vm_area_struct *vma, unsigned long addr,update_mmu_cache_pmd (vma, addr, pmd); } +/* +* FOLL_FORCE can write to even unwritable pmd's, but only +* after we've gone through a COW cycle and they are dirty. +*/ +static inline bool can_follow_write_pmd (pmd_t pmd,unsigned int flags) +{ +return pmd_write (pmd) || + ((flags &FOLL_FORCE ) && (flags &FOLL_COW ) &&pmd_dirty (pmd)); +} +struct page *follow_devmap_pmd (struct vm_area_struct *vma,unsigned long addr,pmd_t *pmd,int flags) {
@@ -783,7 +793,7 @@ struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr,assert_spin_locked (pmd_lockptr (mm, pmd)); -if (flags &FOLL_WRITE && !pmd_write (*pmd)) +if (flags &FOLL_WRITE && !can_follow_write_pmd (*pmd, flags))return NULL ;if (pmd_present (*pmd) &&pmd_devmap (*pmd))
@@ -1137,7 +1147,7 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma,assert_spin_locked (pmd_lockptr (mm, pmd)); -if (flags &FOLL_WRITE && !pmd_write (*pmd)) +if (flags &FOLL_WRITE && !can_follow_write_pmd (*pmd, flags)) goto out;/* Avoid dumping huge zero page */
Signature · 7 lines
2.9.3 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 49 lines · Keno Fischer
In
19be0eaff("mm: removegup_flagsFOLL_WRITEgames from__get_user_pages()"), the mm code was changed from unsettingFOLL_WRITEafter a COW was resolved to setting the (newly introduced)FOLL_COWinstead. Simultaneously, the check ingup.cwas updated to still allow writes withFOLL_FORCEset ifFOLL_COWhad also been set. However, a similar check inhuge_memory.cwas forgotten. As a result, remote memory writes to ro regions of memory backed by transparent huge pages cause an infinite loop in the kernel (handle_mm_faultsetsFOLL_COWand returns 0 causing a retry, butfollow_trans_huge_pmdbails out immidiately because(flags & FOLL_WRITE) && !pmd_write(*pmd)is true. While in this state, the process is stil SIGKILLable, but little else works (e.g. no ptrace attach, no other signals). This is easily reproduced with the following code (assuming thp are set to always):``` #include <assert.h> #include <fcntl.h> #include <stdint.h> #include <stdio.h> #include <string.h> #include <sys/mman.h> #include <sys/stat.h> #include <sys/types.h> #include <sys/wait.h> #include <unistd.h> #define TEST_SIZE 5 * 1024 * 1024 int main(void) { int status; pid_t child; int fd = open("/proc/self/mem", O_RDWR); void *addr = mmap(NULL, TEST_SIZE, PROT_READ, MAP_ANONYMOUS | MAP_PRIVATE, 0, 0); assert(addr != MAP_FAILED); pid_t parent_pid = getpid(); if ((child = fork()) == 0) { void *addr2 = mmap(NULL, TEST_SIZE, PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE, 0, 0); assert(addr2 != MAP_FAILED); memset(addr2, 'a', TEST_SIZE); pwrite(fd, addr2, TEST_SIZE, (uintptr_t)addr); return 0; } assert(child == waitpid(child, &status, 0)); assert(WIFEXITED(status) && WEXITSTATUS(status) == 0); return 0; } ```
Quoted 6 lines
Fix this by updating the instances in
huge_memory.canalogously to the update ingup.cin the original commit. The same pattern existed infollow_devmap_pmd, so I have changed that location as well. However, I do not have a test case that for that code path.Signed-off-by Keno Fischer <[email protected]>
Good catch.
Quoted 26 lines
--- mm/huge_memory.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)mm/huge_memory.c +10
-0@@ -773,6 +773,16 @@ static void touch_pmd(struct vm_area_struct *vma, unsigned long addr,update_mmu_cache_pmd (vma, addr, pmd); } +/* +* FOLL_FORCE can write to even unwritable pmd's, but only +* after we've gone through a COW cycle and they are dirty. +*/ +static inline bool can_follow_write_pmd (pmd_t pmd,unsigned int flags) +{ +return pmd_write (pmd) || + ((flags &FOLL_FORCE ) && (flags &FOLL_COW ) &&pmd_dirty (pmd)); +} +struct page *follow_devmap_pmd (struct vm_area_struct *vma,unsigned long addr,pmd_t *pmd,int flags) {
mm/huge_memory.c struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr, · line 797
Quoted 6 lines
mm/huge_memory.c +1
-1@@ -783,7 +793,7 @@ struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr,assert_spin_locked (pmd_lockptr (mm, pmd)); -if (flags &FOLL_WRITE && !pmd_write (*pmd)) +if (flags &FOLL_WRITE && !can_follow_write_pmd (*pmd, flags))return NULL ;
I don't think this part is needed: once we COW devmap PMD entry, we split it into PTE table, so IIUC we never get here with PMD.
Maybe we should WARN_ONCE() if have FOLL_COW here.
Quoted 19 lines
mm/huge_memory.c +1
-1if (pmd_present (*pmd) &&pmd_devmap (*pmd))@@ -1137,7 +1147,7 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma,assert_spin_locked (pmd_lockptr (mm, pmd)); -if (flags &FOLL_WRITE && !pmd_write (*pmd)) +if (flags &FOLL_WRITE && !can_follow_write_pmd (*pmd, flags)) goto out;/* Avoid dumping huge zero page */ Signature · 7 lines
2.9.3 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Signature · 7 lines
Kirill A. Shutemov -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 4 lines
Quoted 4 lines
mm/huge_memory.c +0
-0@@ -783,7 +793,7 @@ struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr,assert_spin_locked (pmd_lockptr (mm, pmd));
Quoted 8 lines
mm/huge_memory.c struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr, · line 797
Quoted 3 lines
mm/huge_memory.c +1
-1-if (flags &FOLL_WRITE && !pmd_write (*pmd)) +if (flags &FOLL_WRITE && !can_follow_write_pmd (*pmd, flags))return NULL ;I don't think this part is needed: once we COW devmap PMD entry, we split it into PTE table, so IIUC we never get here with PMD.
Maybe we should
WARN_ONCE()if haveFOLL_COWhere.
Sounds good to me. As I said, I don't have a testcase for this code path, I just noticed the same pattern.
Will send an updated patch with the WARN_ONCE there.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 6 lines · Kirill A. Shutemov
Quoted 5 lines · Keno Fischer
struct page *follow_devmap_pmd(struct
vm_area_struct*vma, unsigned long addr,mm/huge_memory.c +0
-0pmd_t *pmd, int flags) {@@ -783,7 +793,7 @@ struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr,
Quoted 8 lines
mm/huge_memory.c struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr, · line 797
Quoted 5 lines
mm/huge_memory.c +1
-1assert_spin_locked (pmd_lockptr (mm, pmd)); -if (flags &FOLL_WRITE && !pmd_write (*pmd)) +if (flags &FOLL_WRITE && !can_follow_write_pmd (*pmd, flags))return NULL ;I don't think this part is needed: once we COW devmap PMD entry, we split it into PTE table, so IIUC we never get here with PMD.
Hi Kirill,
Would you mind double-checking that? You certainly know devmap better than me, but I feel safer with Keno's original as above.
I can see that fs/dax.c dax_iomap_pmd_fault() does
/* Fall back to PTEs if we're going to COW */ if (write && !(vma->vm_flags & VM_SHARED)) goto fallback;
But isn't there a case of O_RDWR fd, VM_SHARED PROT_READ mmap, and FOLL_FORCE write to it, which does not COW (but relies on FOLL_COW)?
Hugh
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 21 lines · Hugh Dickins
Quoted 6 lines · Kirill A. Shutemov
Quoted 5 lines · Keno Fischer
struct page *follow_devmap_pmd(struct
vm_area_struct*vma, unsigned long addr,mm/huge_memory.c +0
-0pmd_t *pmd, int flags) {@@ -783,7 +793,7 @@ struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr,Quoted 8 lines
mm/huge_memory.c struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr, · line 797
Quoted 5 lines
mm/huge_memory.c +1
-1assert_spin_locked (pmd_lockptr (mm, pmd)); -if (flags &FOLL_WRITE && !pmd_write (*pmd)) +if (flags &FOLL_WRITE && !can_follow_write_pmd (*pmd, flags))return NULL ;I don't think this part is needed: once we COW devmap PMD entry, we split it into PTE table, so IIUC we never get here with PMD.
Hi Kirill,
Would you mind double-checking that? You certainly know devmap better than me, but I feel safer with Keno's original as above.
Quoted 8 lines
I can see that
fs/dax.cdax_iomap_pmd_fault()does/* Fall back to PTEs if we're going to COW */ if (write && !(vma->vm_flags & VM_SHARED)) goto fallback;But isn't there a case of
O_RDWRfd,VM_SHAREDPROT_READmmap, andFOLL_FORCEwrite to it, which does not COW (but relies onFOLL_COW)?
And now I think I'm wrong, but please double-check even so: I think that case gets ruled out by the !is_cow_mapping(vm_flags) check in mm/gup.c, where we used to have a WARN_ON_ONCE() for a while.
Hugh
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 28 lines · Hugh Dickins
Quoted 27 lines · Hugh Dickins
Quoted 6 lines · Kirill A. Shutemov
Quoted 5 lines · Keno Fischer
struct page *follow_devmap_pmd(struct
vm_area_struct*vma, unsigned long addr,mm/huge_memory.c +0
-0pmd_t *pmd, int flags) {@@ -783,7 +793,7 @@ struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr,Quoted 8 lines
mm/huge_memory.c struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr, · line 797
Quoted 5 lines
mm/huge_memory.c +1
-1assert_spin_locked (pmd_lockptr (mm, pmd)); -if (flags &FOLL_WRITE && !pmd_write (*pmd)) +if (flags &FOLL_WRITE && !can_follow_write_pmd (*pmd, flags))return NULL ;I don't think this part is needed: once we COW devmap PMD entry, we split it into PTE table, so IIUC we never get here with PMD.
Hi Kirill,
Would you mind double-checking that? You certainly know devmap better than me, but I feel safer with Keno's original as above.
I can see that
fs/dax.cdax_iomap_pmd_fault()does/* Fall back to PTEs if we're going to COW */ if (write && !(vma->vm_flags & VM_SHARED)) goto fallback;
Quoted 6 lines
Quoted 2 lines
But isn't there a case of
O_RDWRfd,VM_SHAREDPROT_READmmap, andFOLL_FORCEwrite to it, which does not COW (but relies onFOLL_COW)?And now I think I'm wrong, but please double-check even so: I think that case gets ruled out by the !is_cow_mapping(vm_flags) check in
mm/gup.c, where we used to have aWARN_ON_ONCE()for a while.
Right, !is_cow_mapping(vm_flags) will filter the case out.
Also there's no way we will get FOLL_COW set for file THP (dax or not): we never return VM_FAULT_WRITE there.
Signature · 7 lines
Kirill A. Shutemov -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>