Infinite loop with DAX PMD faults
Subsystems (2)
Jan Kara
reviewerHello,
When testing my DAX patches rebased on top of Ross' DAX PMD series, I've come across the following issue with generic/344 test from xfstests. The test ends in an infinite fault loop when we fault index 0 over and over again never finishing the fault. The problem is that we do a write fault for index 0 when there is PMD for that index. So we enter wp_huge_pmd(). For whatever reason that returns VM_FAULT_FALLBACK so we continue to handle_pte_fault(). There we do
if (pmd_trans_unstable(vmf->pmd) || pmd_devmap(*vmf->pmd))
check which is true - the PMD we have is pmd_trans_huge() - so we 'return 0' and that results in retrying the fault and all happens from the beginning again.
It isn't quite obvious how to break that cycle to me. The comment before pmd_none_or_trans_huge_or_clear_bad() goes to great lengths explaining possible races when PMD is pmd_trans_huge() so it needs careful evaluation what needs to be done for DAX. Ross, any idea?
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 12 lines · Jan Kara
Hello,
When testing my DAX patches rebased on top of Ross' DAX PMD series, I've come across the following issue with generic/344 test from xfstests. The test ends in an infinite fault loop when we fault index 0 over and over again never finishing the fault. The problem is that we do a write fault for index 0 when there is PMD for that index. So we enter
wp_huge_pmd(). For whatever reason that returnsVM_FAULT_FALLBACKso we continue tohandle_pte_fault(). There we doif (pmd_trans_unstable(vmf->pmd) || pmd_devmap(*vmf->pmd))
Quoted 8 lines
check which is true - the PMD we have is
pmd_trans_huge()- so we 'return 0' and that results in retrying the fault and all happens from the beginning again.It isn't quite obvious how to break that cycle to me. The comment before
pmd_none_or_trans_huge_or_clear_bad()goes to great lengths explaining possible races when PMD ispmd_trans_huge()so it needs careful evaluation what needs to be done for DAX. Ross, any idea?
Can you bisect it with CONFIG_BROKEN removed from older kernels?
I remember tracking down something like this when initially doing the pmd support. It ended up being a missed pmd_devmap() check in the fault path, so it may not be the same issue. It would at least be interesting to see if 4.6 fails in a similar manner with this test and FS_DAX_PMD enabled.
Quoted 22 lines · Dan Williams
Quoted 12 lines · Jan Kara
Hello,
When testing my DAX patches rebased on top of Ross' DAX PMD series, I've come across the following issue with generic/344 test from xfstests. The test ends in an infinite fault loop when we fault index 0 over and over again never finishing the fault. The problem is that we do a write fault for index 0 when there is PMD for that index. So we enter
wp_huge_pmd(). For whatever reason that returnsVM_FAULT_FALLBACKso we continue tohandle_pte_fault(). There we doif (pmd_trans_unstable(vmf->pmd) || pmd_devmap(*vmf->pmd))Quoted 8 lines
check which is true - the PMD we have is
pmd_trans_huge()- so we 'return 0' and that results in retrying the fault and all happens from the beginning again.It isn't quite obvious how to break that cycle to me. The comment before
pmd_none_or_trans_huge_or_clear_bad()goes to great lengths explaining possible races when PMD ispmd_trans_huge()so it needs careful evaluation what needs to be done for DAX. Ross, any idea?
Quoted 7 lines
Can you bisect it with
CONFIG_BROKENremoved from older kernels?I remember tracking down something like this when initially doing the pmd support. It ended up being a missed
pmd_devmap()check in the fault path, so it may not be the same issue. It would at least be interesting to see if 4.6 fails in a similar manner with this test andFS_DAX_PMDenabled.
I've been able to reproduce this with my v4.9-rc2 branch, but it doesn't reproduce with the old v4.6 kernel.
My guess is that this might be because in the old v4.6 kernel, PMD faults don't actually work most of the time because most users don't pass an 2MiB aligned address to mmap. This was fixed by Toshi's patches:
dbe6ec8 ext2/4, xfs: call thp_get_unmapped_area() for pmd mappings 74d2fad thp, dax: add thp_get_unmapped_area for pmd mappings
Anyway, I'm off to try and understand this failure more deeply.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Quoted 30 lines · Ross Zwisler
Quoted 29 lines · Dan Williams
Quoted 19 lines · Jan Kara
Hello,
When testing my DAX patches rebased on top of Ross' DAX PMD series, I've come across the following issue with generic/344 test from xfstests. The test ends in an infinite fault loop when we fault index 0 over and over again never finishing the fault. The problem is that we do a write fault
for index 0 when there is PMD for that index. So we enterwp_huge_pmd(). For whatever reason that returnsVM_FAULT_FALLBACKso we continue tohandle_pte_fault(). There we doif (
pmd_trans_unstable(vmf->pmd)|| pmd_devmap(*vmf-Quoted 1 line
pmd))
check which is true - the PMD we have is
pmd_trans_huge()- so we 'return 0' and that results in retrying the fault and all happens from the beginning again.Quoted 5 lines
It isn't quite obvious how to break that cycle to me. The comment before
pmd_none_or_trans_huge_or_clear_bad()goes to great lengths explaining possible races when PMD ispmd_trans_huge()so it needs careful evaluation what needs to be done for DAX. Ross, any idea?Can you bisect it with
CONFIG_BROKENremoved from older kernels?
Quoted 8 lines
Quoted 5 lines
I remember tracking down something like this when initially doing the pmd support. It ended up being a missed
pmd_devmap()check in the fault path, so it may not be the same issue. It would at least be interesting to see if 4.6 fails in a similar manner with this test andFS_DAX_PMDenabled.I've been able to reproduce this with my v4.9-rc2 branch, but it doesn't reproduce with the old v4.6 kernel.
Not sure if it's relevant, but as FYI I fixed a similar issue before.
commit 59bf4fb9d386601cbaa70a9b00159abb846dedaa
dax: Split pmd map when fallback on COW
-Toshi
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Quoted 40 lines · Toshimitsu
Quoted 30 lines · Ross Zwisler
Quoted 29 lines · Dan Williams
Quoted 19 lines · Jan Kara
Hello,
When testing my DAX patches rebased on top of Ross' DAX PMD series, I've come across the following issue with generic/344 test from xfstests. The test ends in an infinite fault loop when we fault index 0 over and over again never finishing the fault. The problem is that we do a write fault
for index 0 when there is PMD for that index. So we enterwp_huge_pmd(). For whatever reason that returnsVM_FAULT_FALLBACKso we continue tohandle_pte_fault(). There we doif (
pmd_trans_unstable(vmf->pmd)|| pmd_devmap(*vmf-Quoted 1 line
pmd))
check which is true - the PMD we have is
pmd_trans_huge()- so we 'return 0' and that results in retrying the fault and all happens from the beginning again.Quoted 5 lines
It isn't quite obvious how to break that cycle to me. The comment before
pmd_none_or_trans_huge_or_clear_bad()goes to great lengths explaining possible races when PMD ispmd_trans_huge()so it needs careful evaluation what needs to be done for DAX. Ross, any idea?Can you bisect it with
CONFIG_BROKENremoved from older kernels?Quoted 8 lines
Quoted 5 lines
I remember tracking down something like this when initially doing the pmd support. It ended up being a missed
pmd_devmap()check in the fault path, so it may not be the same issue. It would at least be interesting to see if 4.6 fails in a similar manner with this test andFS_DAX_PMDenabled.I've been able to reproduce this with my v4.9-rc2 branch, but it doesn't reproduce with the old v4.6 kernel.
Quoted 6 lines
Not sure if it's relevant, but as FYI I fixed a similar issue before.
commit
59bf4fb9d386601cbaa70a9b00159abb846dedaa
dax: Split pmd map when fallback on COW-Toshi
Thanks! Applying a similar patch solves this deadlock. Unfortunately I don't (yet?) understand this well enough to say whether this is the correct solution, but it makes generic/344 + PMDs pass. :)
Does anyone with more mm knowledge have time to review?
_______________________________________________
Linux-nvdimm mailing list [email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Jan Kara
reviewerQuoted 46 lines · Ross Zwisler
Quoted 45 lines · Toshimitsu
Quoted 30 lines · Ross Zwisler
Quoted 29 lines · Dan Williams
Quoted 19 lines · Jan Kara
Hello,
When testing my DAX patches rebased on top of Ross' DAX PMD series, I've come across the following issue with generic/344 test from xfstests. The test ends in an infinite fault loop when we fault index 0 over and over again never finishing the fault. The problem is that we do a write fault
for index 0 when there is PMD for that index. So we enterwp_huge_pmd(). For whatever reason that returnsVM_FAULT_FALLBACKso we continue tohandle_pte_fault(). There we doif (
pmd_trans_unstable(vmf->pmd)|| pmd_devmap(*vmf-Quoted 1 line
pmd))
check which is true - the PMD we have is
pmd_trans_huge()- so we 'return 0' and that results in retrying the fault and all happens from the beginning again.Quoted 5 lines
It isn't quite obvious how to break that cycle to me. The comment before
pmd_none_or_trans_huge_or_clear_bad()goes to great lengths explaining possible races when PMD ispmd_trans_huge()so it needs careful evaluation what needs to be done for DAX. Ross, any idea?Can you bisect it with
CONFIG_BROKENremoved from older kernels?Quoted 8 lines
Quoted 5 lines
I remember tracking down something like this when initially doing the pmd support. It ended up being a missed
pmd_devmap()check in the fault path, so it may not be the same issue. It would at least be interesting to see if 4.6 fails in a similar manner with this test andFS_DAX_PMDenabled.I've been able to reproduce this with my v4.9-rc2 branch, but it doesn't reproduce with the old v4.6 kernel.
Not sure if it's relevant, but as FYI I fixed a similar issue before.
commit
59bf4fb9d386601cbaa70a9b00159abb846dedaa
dax: Split pmd map when fallback on COW
Quoted 7 lines
Quoted 1 line
-Toshi
Thanks! Applying a similar patch solves this deadlock. Unfortunately I don't (yet?) understand this well enough to say whether this is the correct solution, but it makes generic/344 + PMDs pass. :)
Does anyone with more mm knowledge have time to review?
I'm not really much into huge pages but AFAICT that should fix the problem. I'm just not sure whether in other cases when we return VM_FAULT_FALLBACK we don't need something similar. Probably this will need some experiments ;).
Honza
Signature · 6 lines
Jan Kara <[email protected]> SUSE Labs, CR _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
Quoted 66 lines · Jan Kara
Quoted 65 lines · Ross Zwisler
Quoted 50 lines · Toshimitsu
Quoted 45 lines · Ross Zwisler
Quoted 36 lines · Dan Williams
Quoted 23 lines · Jan Kara
Hello,
When testing my DAX patches rebased on top of Ross' DAX PMD series, I've come across the following issue with generic/344 test from xfstests. The test ends in an infinite fault loop when we fault index 0 over and over again never finishing the fault. The problem is that we do a write fault for index 0 when there is PMD for that index. So we enter
wp_huge_pmd(). For whatever reason that returnsVM_FAULT_FALLBACKso we continue tohandle_pte_fault(). There we doif (
pmd_trans_unstable(vmf->pmd)||
pmd_devmap(*vmf-Quoted 2 lines
pmd))
check which is true - the PMD we have is
pmd_trans_huge()- so we 'return 0' and that results in retrying the fault and all happens from the beginning again.Quoted 5 lines
It isn't quite obvious how to break that cycle to me. The comment before
pmd_none_or_trans_huge_or_clear_bad()goes to great lengths explaining possible races when PMD ispmd_trans_huge()so it needs careful evaluation what needs to be done for DAX. Ross, any idea?Can you bisect it with
CONFIG_BROKENremoved from older kernels?Quoted 6 lines
I remember tracking down something like this when initially doing the pmd support. It ended up being a missed
pmd_devmap()check in the fault path, so it may not be the same issue. It would at least be interesting to see if 4.6 fails in a similar manner with this test andFS_DAX_PMDenabled.Quoted 2 lines
I've been able to reproduce this with my v4.9-rc2 branch, but it doesn't reproduce with the old v4.6 kernel.
Quoted 7 lines
Not sure if it's relevant, but as FYI I fixed a similar issue before.
commit
59bf4fb9d386601cbaa70a9b00159abb846dedaa
dax: Split pmd map when fallback on COW-Toshi
Thanks! Applying a similar patch solves this
deadlock. Unfortunately I don't (yet?) understand this well enough to say whether this is the correct solution, but it makes generic/344 + PMDs pass. :)
Quoted 7 lines
Quoted 1 line
Does anyone with more mm knowledge have time to review?
I'm not really much into huge pages but AFAICT that should fix the problem. I'm just not sure whether in other cases when we return
VM_FAULT_FALLBACKwe don't need something similar. Probably this will need some experiments
;).
Good to know it worked. :-) I think we need to split a pmd mapping in the case of COW fallback. The pte handler may not proceed when a pmd mapping is still in-place.
Thanks,
-Toshi
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Jan Kara
reviewerQuoted 17 lines · Dan Williams
Quoted 16 lines · Jan Kara
Hello,
When testing my DAX patches rebased on top of Ross' DAX PMD series, I've come across the following issue with generic/344 test from xfstests. The test ends in an infinite fault loop when we fault index 0 over and over again never finishing the fault. The problem is that we do a write fault for index 0 when there is PMD for that index. So we enter
wp_huge_pmd(). For whatever reason that returnsVM_FAULT_FALLBACKso we continue tohandle_pte_fault(). There we doif (pmd_trans_unstable(vmf->pmd) || pmd_devmap(*vmf->pmd))check which is true - the PMD we have is
pmd_trans_huge()- so we 'return 0' and that results in retrying the fault and all happens from the beginning again.
Quoted 6 lines
Quoted 4 lines
It isn't quite obvious how to break that cycle to me. The comment before
pmd_none_or_trans_huge_or_clear_bad()goes to great lengths explaining possible races when PMD ispmd_trans_huge()so it needs careful evaluation what needs to be done for DAX. Ross, any idea?Can you bisect it with
CONFIG_BROKENremoved from older kernels?
I can try (but likely won't get to it before Kernel Summit, not sure if I'll have time for that there).
Quoted 5 lines
I remember tracking down something like this when initially doing the pmd support. It ended up being a missed
pmd_devmap()check in the fault path, so it may not be the same issue. It would at least be interesting to see if 4.6 fails in a similar manner with this test andFS_DAX_PMDenabled.
BTW, the results of checks for the PMD are:
pmd_devmap(*vmf->pmd) == 0pmd_trans_huge(*vmf->pmd) == 1pmd_bad(*vmf->pmd) == 1
I'll see if I can get any meaningful test running based on 4.6...
Honza
Signature · 6 lines
Jan Kara <[email protected]> SUSE Labs, CR _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
Quoted 12 lines · Jan Kara
Hello,
When testing my DAX patches rebased on top of Ross' DAX PMD series, I've come across the following issue with generic/344 test from xfstests. The test ends in an infinite fault loop when we fault index 0 over and over again never finishing the fault. The problem is that we do a write fault for index 0 when there is PMD for that index. So we enter
wp_huge_pmd(). For whatever reason that returnsVM_FAULT_FALLBACKso we continue tohandle_pte_fault(). There we doif (pmd_trans_unstable(vmf->pmd) || pmd_devmap(*vmf->pmd))
Quoted 8 lines
check which is true - the PMD we have is
pmd_trans_huge()- so we 'return 0' and that results in retrying the fault and all happens from the beginning again.It isn't quite obvious how to break that cycle to me. The comment before
pmd_none_or_trans_huge_or_clear_bad()goes to great lengths explaining possible races when PMD ispmd_trans_huge()so it needs careful evaluation what needs to be done for DAX. Ross, any idea?
I'll try & reproduce this, and I'll get back to you.
Jan Kara
reviewerQuoted 15 lines · Ross Zwisler
Quoted 14 lines · Jan Kara
When testing my DAX patches rebased on top of Ross' DAX PMD series, I've come across the following issue with generic/344 test from xfstests. The test ends in an infinite fault loop when we fault index 0 over and over again never finishing the fault. The problem is that we do a write fault for index 0 when there is PMD for that index. So we enter
wp_huge_pmd(). For whatever reason that returnsVM_FAULT_FALLBACKso we continue tohandle_pte_fault(). There we doif (pmd_trans_unstable(vmf->pmd) || pmd_devmap(*vmf->pmd))check which is true - the PMD we have is
pmd_trans_huge()- so we 'return 0' and that results in retrying the fault and all happens from the beginning again.
Quoted 6 lines
Quoted 4 lines
It isn't quite obvious how to break that cycle to me. The comment before
pmd_none_or_trans_huge_or_clear_bad()goes to great lengths explaining possible races when PMD ispmd_trans_huge()so it needs careful evaluation what needs to be done for DAX. Ross, any idea?I'll try & reproduce this, and I'll get back to you.
For me it happened with ext4 which returned VM_FAULT_FALLBACK from its pmd_fault handler on write fault (likely we were not able to allocate sufficiently large contiguous hunk). So I'm not sure you will be able to easily reproduce just with your series. However tweaking XFS to return VM_FAULT_FALLBACK when FAULT_FLAG_WRITE is set should do the trick.
Honza
Signature · 6 lines
Jan Kara <[email protected]> SUSE Labs, CR _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
Quoted 21 lines · Jan Kara
Quoted 20 lines · Ross Zwisler
Quoted 18 lines · Jan Kara
When testing my DAX patches rebased on top of Ross' DAX PMD series, I've come across the following issue with generic/344 test from xfstests. The test ends in an infinite fault loop when we fault index 0 over and over again never finishing the fault. The problem is that we do a write fault for index 0 when there is PMD for that index. So we enter
wp_huge_pmd(). For whatever reason that returnsVM_FAULT_FALLBACKso we continue tohandle_pte_fault(). There we doif (pmd_trans_unstable(vmf->pmd) || pmd_devmap(*vmf->pmd))check which is true - the PMD we have is
pmd_trans_huge()- so we 'return 0' and that results in retrying the fault and all happens from the beginning again.It isn't quite obvious how to break that cycle to me. The comment before
pmd_none_or_trans_huge_or_clear_bad()goes to great lengths explaining possible races when PMD ispmd_trans_huge()so it needs careful evaluation what needs to be done for DAX. Ross, any idea?
Quoted 7 lines
Quoted 1 line
I'll try & reproduce this, and I'll get back to you.
For me it happened with ext4 which returned
VM_FAULT_FALLBACKfrom itspmd_faulthandler on write fault (likely we were not able to allocate sufficiently large contiguous hunk). So I'm not sure you will be able to easily reproduce just with your series. However tweaking XFS to returnVM_FAULT_FALLBACKwhenFAULT_FLAG_WRITEis set should do the trick.
I was able to reproduce it with XFS, with just my series.
The fallback check that's failing for me is this one:
if (pfn_t_to_pfn(dax.pfn) & PG_PMD_COLOUR)
in dax_pmd_insert_mapping().
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm