arm64/mm: pmd_mkinvalid() must handle swap pmds
Subsystems (3)
PATCH v2 1/1 arm64/mm: pmd_mkinvalid() must handle swap pmds
__split_huge_pmd_locked() can be called for a present THP, devmap or (non-present) migration entry. It calls pmdp_invalidate() unconditionally on the pmdp and only determines if it is present or not based on the returned old pmd.
But arm64's pmd_mkinvalid(), called by pmdp_invalidate(), unconditionally sets the PMD_PRESENT_INVALID flag, which causes future pmd_present() calls to return true - even for a swap pmd. Therefore any lockless pgtable walker could see the migration entry pmd in this state and start interpretting the fields (e.g. pmd_pfn()) as if it were present, leading to BadThings (TM). GUP-fast appears to be one such lockless pgtable walker.
While the obvious fix is for core-mm to avoid such calls for non-present pmds (pmdp_invalidate() will also issue TLBI which is not necessary for this case either), all other arches that implement pmd_mkinvalid() do it in such a way that it is robust to being called with a non-present pmd. So it is simpler and safer to make arm64 robust too. This approach means we can even add tests to debug_vm_pgtable.c to validate the required behaviour.
This is a theoretical bug found during code review. I don't have any test case to trigger it in practice.
---
Hi all,
v1 of this fix [1] took the approach of fixing core-mm to never call pmdp_invalidate() on a non-present pmd. But Zi Yan highlighted that only arm64 suffers this problem; all other arches are robust. So his suggestion was to instead make arm64 robust in the same way and add tests to validate it. Despite my stated reservations in the context of the v1 discussion, having thought on it for a bit, I now agree with Zi Yan. Hence this post.
Andrew has v1 in mm-unstable at the moment, so probably the best thing to do is remove it from there and have this go in through the arm64 tree? Assuming there is agreement that this approach is right one.
This applies on top of v6.9-rc5. Passes all the mm selftests on arm64.
[1] https://lore.kernel.org/linux-mm/[email protected]/
Thanks,
Ryan
arch/arm64/include/asm/pgtable.h | 12 +++++-- mm/debug_vm_pgtable.c | 61 ++++++++++++++++++++++++++++++++ 2 files changed, 71 insertions(+), 2 deletions(-)
arch/arm64/include/asm/pgtable.h +10 -2
@@ -511,8 +511,16 @@ static inline int pmd_trans_huge(pmd_t pmd)static inline pmd_t pmd_mkinvalid (pmd_t pmd) { - pmd =set_pmd_bit (pmd,__pgprot (PMD_PRESENT_INVALID )); - pmd =clear_pmd_bit (pmd,__pgprot (PMD_SECT_VALID )); +/* +* If not valid then either we are already present-invalid or we are +* not-present (i.e. none or swap entry). We must not convert +* not-present to present-invalid. Unbelievably, the core-mm may call +* pmd_mkinvalid() for a swap entry and all other arches can handle it. +*/ +if (pmd_valid (pmd)) { + pmd =set_pmd_bit (pmd,__pgprot (PMD_PRESENT_INVALID )); + pmd =clear_pmd_bit (pmd,__pgprot (PMD_SECT_VALID )); + }return pmd; }
mm/debug_vm_pgtable.c +61 -0
73 lines not on this page · raw message
--
2.25.1
Quoted 67 lines · Ryan Roberts
__split_huge_pmd_locked()can be called for a present THP, devmap or (non-present) migration entry. It callspmdp_invalidate()unconditionally on the pmdp and only determines if it is present or not based on the returned old pmd.But arm64's
pmd_mkinvalid(), called bypmdp_invalidate(), unconditionally sets thePMD_PRESENT_INVALIDflag, which causes futurepmd_present()calls to return true - even for a swap pmd. Therefore any lockless pgtable walker could see the migration entry pmd in this state and start interpretting the fields (e.g.pmd_pfn()) as if it were present, leading to BadThings (TM). GUP-fast appears to be one such lockless pgtable walker.While the obvious fix is for core-mm to avoid such calls for non-present pmds (
pmdp_invalidate()will also issue TLBI which is not necessary for this case either), all other arches that implementpmd_mkinvalid()do it in such a way that it is robust to being called with a non-present pmd. So it is simpler and safer to make arm64 robust too. This approach means we can even add tests todebug_vm_pgtable.cto validate the required behaviour.This is a theoretical bug found during code review. I don't have any test case to trigger it in practice.
Fixes 53fa117bb33c ("arm64/mm: Enable THP migration")Signed-off-by Ryan Roberts <[email protected]>---
Hi all,
v1 of this fix [1] took the approach of fixing core-mm to never call
pmdp_invalidate()on a non-present pmd. But Zi Yan highlighted that only arm64 suffers this problem; all other arches are robust. So his suggestion was to instead make arm64 robust in the same way and add tests to validate it. Despite my stated reservations in the context of the v1 discussion, having thought on it for a bit, I now agree with Zi Yan. Hence this post.Andrew has v1 in mm-unstable at the moment, so probably the best thing to do is remove it from there and have this go in through the arm64 tree? Assuming there is agreement that this approach is right one.
This applies on top of v6.9-rc5. Passes all the mm selftests on arm64.
[1] https://lore.kernel.org/linux-mm/[email protected]/
Thanks,
Ryanarch/arm64/include/asm/pgtable.h | 12 +++++-- mm/debug_vm_pgtable.c | 61 ++++++++++++++++++++++++++++++++ 2 files changed, 71 insertions(+), 2 deletions(-)arch/arm64/include/asm/pgtable.h +3
-2@@ -511,8 +511,16 @@ static inline int pmd_trans_huge(pmd_t pmd)static inline pmd_t pmd_mkinvalid (pmd_t pmd) { - pmd =set_pmd_bit (pmd,__pgprot (PMD_PRESENT_INVALID )); - pmd =clear_pmd_bit (pmd,__pgprot (PMD_SECT_VALID )); +/* +* If not valid then either we are already present-invalid or we are +* not-present (i.e. none or swap entry). We must not convert
arch/arm64/include/asm/pgtable.h static inline int pmd_trans_huge(pmd_t pmd) · line 526
Quoted 10 lines
arch/arm64/include/asm/pgtable.h +7
-0+* not-present to present-invalid. Unbelievably, the core-mm may call +* pmd_mkinvalid() for a swap entry and all other arches can handle it. +*/ +if (pmd_valid (pmd)) { + pmd =set_pmd_bit (pmd,__pgprot (PMD_PRESENT_INVALID )); + pmd =clear_pmd_bit (pmd,__pgprot (PMD_SECT_VALID )); + }return pmd; }
But it might be worth splitting the tests from the fix to make backporting easier.
Catalin -- I assume you'll pick this up, but please shout if you want me to take it instead.
Will
Quoted 76 lines · Will Deacon
Quoted 75 lines · Ryan Roberts
__split_huge_pmd_locked()can be called for a present THP, devmap or (non-present) migration entry. It callspmdp_invalidate()unconditionally on the pmdp and only determines if it is present or not based on the returned old pmd.But arm64's
pmd_mkinvalid(), called bypmdp_invalidate(), unconditionally sets thePMD_PRESENT_INVALIDflag, which causes futurepmd_present()calls to return true - even for a swap pmd. Therefore any lockless pgtable walker could see the migration entry pmd in this state and start interpretting the fields (e.g.pmd_pfn()) as if it were present, leading to BadThings (TM). GUP-fast appears to be one such lockless pgtable walker.While the obvious fix is for core-mm to avoid such calls for non-present pmds (
pmdp_invalidate()will also issue TLBI which is not necessary for this case either), all other arches that implementpmd_mkinvalid()do it in such a way that it is robust to being called with a non-present pmd. So it is simpler and safer to make arm64 robust too. This approach means we can even add tests todebug_vm_pgtable.cto validate the required behaviour.This is a theoretical bug found during code review. I don't have any test case to trigger it in practice.
Fixes 53fa117bb33c ("arm64/mm: Enable THP migration")Signed-off-by Ryan Roberts <[email protected]>---
Hi all,
v1 of this fix [1] took the approach of fixing core-mm to never call
pmdp_invalidate()on a non-present pmd. But Zi Yan highlighted that only arm64 suffers this problem; all other arches are robust. So his suggestion was to instead make arm64 robust in the same way and add tests to validate it. Despite my stated reservations in the context of the v1 discussion, having thought on it for a bit, I now agree with Zi Yan. Hence this post.Andrew has v1 in mm-unstable at the moment, so probably the best thing to do is remove it from there and have this go in through the arm64 tree? Assuming there is agreement that this approach is right one.
This applies on top of v6.9-rc5. Passes all the mm selftests on arm64.
[1] https://lore.kernel.org/linux-mm/[email protected]/
Thanks,
Ryanarch/arm64/include/asm/pgtable.h | 12 +++++-- mm/debug_vm_pgtable.c | 61 ++++++++++++++++++++++++++++++++ 2 files changed, 71 insertions(+), 2 deletions(-)arch/arm64/include/asm/pgtable.h +10
-2@@ -511,8 +511,16 @@ static inline int pmd_trans_huge(pmd_t pmd)static inline pmd_t pmd_mkinvalid (pmd_t pmd) { - pmd =set_pmd_bit (pmd,__pgprot (PMD_PRESENT_INVALID )); - pmd =clear_pmd_bit (pmd,__pgprot (PMD_SECT_VALID )); +/* +* If not valid then either we are already present-invalid or we are +* not-present (i.e. none or swap entry). We must not convert +* not-present to present-invalid. Unbelievably, the core-mm may call +* pmd_mkinvalid() for a swap entry and all other arches can handle it. +*/ +if (pmd_valid (pmd)) { + pmd =set_pmd_bit (pmd,__pgprot (PMD_PRESENT_INVALID )); + pmd =clear_pmd_bit (pmd,__pgprot (PMD_SECT_VALID )); + }
Quoted 4 lines
arch/arm64/include/asm/pgtable.h static inline int pmd_trans_huge(pmd_t pmd) · line 526
Quoted 2 lines
arch/arm64/include/asm/pgtable.h +0
-0return pmd; }Acked-by Will Deacon <[email protected]>
Thanks
Quoted 3 lines
But it might be worth splitting the tests from the fix to make backporting easier.
Yes good point. I'll leave this hanging for today to see if any more comments come in, and will re-post tomorrow as 2 patches. I assume we need to go fast to catch 6.9.
Quoted 5 lines
Catalin -- I assume you'll pick this up, but please shout if you want me to take it instead.
Will
Catalin Marinas
maintainerQuoted 7 lines · Ryan Roberts
Quoted 2 lines · Will Deacon
But it might be worth splitting the tests from the fix to make backporting easier.
Yes good point. I'll leave this hanging for today to see if any more comments come in, and will re-post tomorrow as 2 patches. I assume we need to go fast to catch 6.9.
Yes, I'll pick it up for 6.9. I can drop the tests from the patch (and their mention in the log) and you can post your tests separately to go via Andrew's tree.
Signature · 1 line
Catalin
Quoted 5 lines · Catalin Marinas
Quoted 4 lines · Ryan Roberts
Quoted 2 lines · Will Deacon
But it might be worth splitting the tests from the fix to make backporting easier.
Quoted 8 lines
Quoted 3 lines
Yes good point. I'll leave this hanging for today to see if any more comments come in, and will re-post tomorrow as 2 patches. I assume we need to go fast to catch 6.9.
Yes, I'll pick it up for 6.9. I can drop the tests from the patch (and their mention in the log) and you can post your tests separately to go via Andrew's tree.
Yep that works for me! thanks.
Quoted 46 lines · Ryan Roberts
__split_huge_pmd_locked()can be called for a present THP, devmap or (non-present) migration entry. It callspmdp_invalidate()unconditionally on the pmdp and only determines if it is present or not based on the returned old pmd.But arm64's
pmd_mkinvalid(), called bypmdp_invalidate(), unconditionally sets thePMD_PRESENT_INVALIDflag, which causes futurepmd_present()calls to return true - even for a swap pmd. Therefore any lockless pgtable walker could see the migration entry pmd in this state and start interpretting the fields (e.g.pmd_pfn()) as if it were present, leading to BadThings (TM). GUP-fast appears to be one such lockless pgtable walker.While the obvious fix is for core-mm to avoid such calls for non-present pmds (
pmdp_invalidate()will also issue TLBI which is not necessary for this case either), all other arches that implementpmd_mkinvalid()do it in such a way that it is robust to being called with a non-present pmd. So it is simpler and safer to make arm64 robust too. This approach means we can even add tests todebug_vm_pgtable.cto validate the required behaviour.This is a theoretical bug found during code review. I don't have any test case to trigger it in practice.
Fixes 53fa117bb33c ("arm64/mm: Enable THP migration")Signed-off-by Ryan Roberts <[email protected]>---
Hi all,
v1 of this fix [1] took the approach of fixing core-mm to never call
pmdp_invalidate()on a non-present pmd. But Zi Yan highlighted that only arm64 suffers this problem; all other arches are robust. So his suggestion was to instead make arm64 robust in the same way and add tests to validate it. Despite my stated reservations in the context of the v1 discussion, having thought on it for a bit, I now agree with Zi Yan. Hence this post.Andrew has v1 in mm-unstable at the moment, so probably the best thing to do is remove it from there and have this go in through the arm64 tree? Assuming there is agreement that this approach is right one.
This applies on top of v6.9-rc5. Passes all the mm selftests on arm64.
Quoted 8 lines
Thanks,
Ryanarch/arm64/include/asm/pgtable.h | 12 +++++-- mm/debug_vm_pgtable.c | 61 ++++++++++++++++++++++++++++++++ 2 files changed, 71 insertions(+), 2 deletions(-)
LGTM. Thanks. Feel free to add Reviewed-by: Zi Yan <[email protected]>.
--
Best Regards,
Yan, Zi
Quoted 13 lines · Ryan Roberts
__split_huge_pmd_locked()can be called for a present THP, devmap or (non-present) migration entry. It callspmdp_invalidate()unconditionally on the pmdp and only determines if it is present or not based on the returned old pmd.But arm64's
pmd_mkinvalid(), called bypmdp_invalidate(), unconditionally sets thePMD_PRESENT_INVALIDflag, which causes futurepmd_present()calls to return true - even for a swap pmd. Therefore any lockless pgtable walker could see the migration entry pmd in this state and start interpretting the fields (e.g.pmd_pfn()) as if it were present, leading to BadThings (TM). GUP-fast appears to be one such lockless pgtable walker.
Quoted 1 line
[...]
Applied to arm64 (for-next/fixes), thanks! It should land in 6.9-rc7. I removed the debug/test code, please send it as a separate patch for 6.10.
[1/1] arm64/mm: pmd_mkinvalid() must handle swap pmds
https://git.kernel.org/arm64/c/e783331c7720
Signature · 1 line
Catalin
Quoted 14 lines · Catalin Marinas
Quoted 13 lines · Ryan Roberts
__split_huge_pmd_locked()can be called for a present THP, devmap or (non-present) migration entry. It callspmdp_invalidate()unconditionally on the pmdp and only determines if it is present or not based on the returned old pmd.But arm64's
pmd_mkinvalid(), called bypmdp_invalidate(), unconditionally sets thePMD_PRESENT_INVALIDflag, which causes futurepmd_present()calls to return true - even for a swap pmd. Therefore any lockless pgtable walker could see the migration entry pmd in this state and start interpretting the fields (e.g.pmd_pfn()) as if it were present, leading to BadThings (TM). GUP-fast appears to be one such lockless pgtable walker.
Quoted 5 lines
Quoted 1 line
[...]
Applied to arm64 (for-next/fixes), thanks! It should land in 6.9-rc7. I removed the debug/test code, please send it as a separate patch for 6.10.
Thanks Catalin! I'm guessing this will turn up in today's linux-next, so if I send the tests today and Andrew puts them straight in mm-unstable (which will goto linux-next) there is no risk that the tests are there without the fix? Or do I need to hold off until the fix is in v6.9-rc7?
Quoted 4 lines
[1/1] arm64/mm:
pmd_mkinvalid()must handle swap pmdshttps://git.kernel.org/arm64/c/e783331c7720
Catalin Marinas
maintainerQuoted 17 lines · Ryan Roberts
Quoted 16 lines · Catalin Marinas
Quoted 14 lines · Ryan Roberts
__split_huge_pmd_locked()can be called for a present THP, devmap or (non-present) migration entry. It callspmdp_invalidate()unconditionally on the pmdp and only determines if it is present or not based on the returned old pmd.But arm64's
pmd_mkinvalid(), called bypmdp_invalidate(), unconditionally sets thePMD_PRESENT_INVALIDflag, which causes futurepmd_present()calls to return true - even for a swap pmd. Therefore any lockless pgtable walker could see the migration entry pmd in this state and start interpretting the fields (e.g.pmd_pfn()) as if it were present, leading to BadThings (TM). GUP-fast appears to be one such lockless pgtable walker.[...]
Quoted 8 lines
Quoted 3 lines
Applied to arm64 (for-next/fixes), thanks! It should land in 6.9-rc7. I removed the debug/test code, please send it as a separate patch for 6.10.
Thanks Catalin! I'm guessing this will turn up in today's linux-next, so if I send the tests today and Andrew puts them straight in mm-unstable (which will goto linux-next) there is no risk that the tests are there without the fix? Or do I need to hold off until the fix is in v6.9-rc7?
It looks like we don't push for-next/fixes to linux-next, it's short-lived usually, it ends up upstream quickly. I can send the pull request later today, should turn up in mainline by tomorrow. You can add a note to your patch for Andrew that it will fail on arm64 until the fix ends up upstream. It's a matter of a couple of days anyway.
Signature · 1 line
Catalin
Quoted 27 lines · Catalin Marinas
Quoted 17 lines · Ryan Roberts
Quoted 16 lines · Catalin Marinas
Quoted 14 lines · Ryan Roberts
__split_huge_pmd_locked()can be called for a present THP, devmap or (non-present) migration entry. It callspmdp_invalidate()unconditionally on the pmdp and only determines if it is present or not based on the returned old pmd.But arm64's
pmd_mkinvalid(), called bypmdp_invalidate(), unconditionally sets thePMD_PRESENT_INVALIDflag, which causes futurepmd_present()calls to return true - even for a swap pmd. Therefore any lockless pgtable walker could see the migration entry pmd in this state and start interpretting the fields (e.g.pmd_pfn()) as if it were present, leading to BadThings (TM). GUP-fast appears to be one such lockless pgtable walker.[...]
Quoted 8 lines
Quoted 3 lines
Applied to arm64 (for-next/fixes), thanks! It should land in 6.9-rc7. I removed the debug/test code, please send it as a separate patch for 6.10.
Thanks Catalin! I'm guessing this will turn up in today's linux-next, so if I send the tests today and Andrew puts them straight in mm-unstable (which will goto linux-next) there is no risk that the tests are there without the fix? Or do I need to hold off until the fix is in v6.9-rc7?
Quoted 5 lines
It looks like we don't push for-next/fixes to linux-next, it's short-lived usually, it ends up upstream quickly. I can send the pull request later today, should turn up in mainline by tomorrow. You can add a note to your patch for Andrew that it will fail on arm64 until the fix ends up upstream. It's a matter of a couple of days anyway.
OK, I just didn't want to send it only for our CI to start failing. I'll do as you suggest.
Catalin Marinas
maintainerQuoted 14 lines · Catalin Marinas
Quoted 13 lines · Ryan Roberts
__split_huge_pmd_locked()can be called for a present THP, devmap or (non-present) migration entry. It callspmdp_invalidate()unconditionally on the pmdp and only determines if it is present or not based on the returned old pmd.But arm64's
pmd_mkinvalid(), called bypmdp_invalidate(), unconditionally sets thePMD_PRESENT_INVALIDflag, which causes futurepmd_present()calls to return true - even for a swap pmd. Therefore any lockless pgtable walker could see the migration entry pmd in this state and start interpretting the fields (e.g.pmd_pfn()) as if it were present, leading to BadThings (TM). GUP-fast appears to be one such lockless pgtable walker.
Quoted 8 lines
Quoted 1 line
[...]
Applied to arm64 (for-next/fixes), thanks! It should land in 6.9-rc7. I removed the debug/test code, please send it as a separate patch for 6.10.
[1/1] arm64/mm:
pmd_mkinvalid()must handle swap pmdshttps://git.kernel.org/arm64/c/e783331c7720
Since Andrew merged the generic mm fix, I dropped this patch from the arm64 for-next/fixes branch.
Signature · 1 line
Catalin
Zi Yan, I'm hoping you might have some input on the below...
Quoted 13 lines · Ryan Roberts
__split_huge_pmd_locked()can be called for a present THP, devmap or (non-present) migration entry. It callspmdp_invalidate()unconditionally on the pmdp and only determines if it is present or not based on the returned old pmd.But arm64's
pmd_mkinvalid(), called bypmdp_invalidate(), unconditionally sets thePMD_PRESENT_INVALIDflag, which causes futurepmd_present()calls to return true - even for a swap pmd. Therefore any lockless pgtable walker could see the migration entry pmd in this state and start interpretting the fields (e.g.pmd_pfn()) as if it were present, leading to BadThings (TM). GUP-fast appears to be one such lockless pgtable walker.
Quoted 4 lines
While the obvious fix is for core-mm to avoid such calls for non-present pmds (
pmdp_invalidate()will also issue TLBI which is not necessary for this case either), all other arches that implementpmd_mkinvalid()do it in such a way that it is robust to being called with a non-present pmd.
OK the plot thickens; The tests I wrote to check that pmd_mkinvalid() is safe for swap entries fails on x86_64. See below...
Quoted 65 lines
So it is simpler and safer to make arm64 robust too. This approach means we can even add tests to
debug_vm_pgtable.cto validate the required behaviour.This is a theoretical bug found during code review. I don't have any test case to trigger it in practice.
Fixes 53fa117bb33c ("arm64/mm: Enable THP migration")Signed-off-by Ryan Roberts <[email protected]>---
Hi all,
v1 of this fix [1] took the approach of fixing core-mm to never call
pmdp_invalidate()on a non-present pmd. But Zi Yan highlighted that only arm64 suffers this problem; all other arches are robust. So his suggestion was to instead make arm64 robust in the same way and add tests to validate it. Despite my stated reservations in the context of the v1 discussion, having thought on it for a bit, I now agree with Zi Yan. Hence this post.Andrew has v1 in mm-unstable at the moment, so probably the best thing to do is remove it from there and have this go in through the arm64 tree? Assuming there is agreement that this approach is right one.
This applies on top of v6.9-rc5. Passes all the mm selftests on arm64.
[1] https://lore.kernel.org/linux-mm/[email protected]/
Thanks,
Ryanarch/arm64/include/asm/pgtable.h | 12 +++++-- mm/debug_vm_pgtable.c | 61 ++++++++++++++++++++++++++++++++ 2 files changed, 71 insertions(+), 2 deletions(-)arch/arm64/include/asm/pgtable.h +10
-2@@ -511,8 +511,16 @@ static inline int pmd_trans_huge(pmd_t pmd)static inline pmd_t pmd_mkinvalid (pmd_t pmd) { - pmd =set_pmd_bit (pmd,__pgprot (PMD_PRESENT_INVALID )); - pmd =clear_pmd_bit (pmd,__pgprot (PMD_SECT_VALID )); +/* +* If not valid then either we are already present-invalid or we are +* not-present (i.e. none or swap entry). We must not convert +* not-present to present-invalid. Unbelievably, the core-mm may call +* pmd_mkinvalid() for a swap entry and all other arches can handle it. +*/ +if (pmd_valid (pmd)) { + pmd =set_pmd_bit (pmd,__pgprot (PMD_PRESENT_INVALID )); + pmd =clear_pmd_bit (pmd,__pgprot (PMD_SECT_VALID )); + }return pmd; }mm/debug_vm_pgtable.c +0
-0
mm/debug_vm_pgtable.c static void __init hugetlb_basic_tests(struct pgtable_debug_args *args) { } · line 961
Quoted 6 lines
mm/debug_vm_pgtable.c +3
-0@@ -956,6 +956,65 @@ static void __init hugetlb_basic_tests(struct pgtable_debug_args *args) { }#endif /* CONFIG_HUGETLB_PAGE */ #ifdef CONFIG_TRANSPARENT_HUGEPAGE +#if !defined(__HAVE_ARCH_PMDP_INVALIDATE) && defined(CONFIG_ARCH_ENABLE_THP_MIGRATION) +static void __init swp_pmd_mkinvalid_tests(struct pgtable_debug_args *args) +{
Printing various values at different locations in this function for debug:
Quoted 6 lines
mm/debug_vm_pgtable.c +6
-0+unsigned long max_swap_offset; +swp_entry_t swp_set, swp_clear, swp_convert; +pmd_t pmd_set, pmd_clear; + +/* +* See generic_max_swapfile_size(): probe the maximum offset, then
mm/debug_vm_pgtable.c static void __init hugetlb_basic_tests(struct pgtable_debug_args *args) { } · line 977
Quoted 10 lines
mm/debug_vm_pgtable.c +10
-0+* create swap entry will all possible bits set and a swap entry will +* all bits clear. +*/ + max_swap_offset =swp_offset (pmd_to_swp_entry (swp_entry_to_pmd (swp_entry (0 , ~0UL )))); + swp_set =swp_entry ((1 <<MAX_SWAPFILES_SHIFT ) -1 , max_swap_offset); + swp_clear =swp_entry (0 ,0 ); + +/* Convert to pmd. */ + pmd_set =swp_entry_to_pmd (swp_set); + pmd_clear =swp_entry_to_pmd (swp_clear);
[ 0.702163] debug_vm_pgtable: [swp_pmd_mkinvalid_tests ]: valid: pmd_set=f800000000000000, pmd_clear=7fffffffffffe00
Quoted 9 lines
mm/debug_vm_pgtable.c +9
-0+ +/* +* Sanity check that the pmds are not-present, not-huge and swap entry +* is recoverable without corruption. +*/ +WARN_ON (pmd_present (pmd_set)); +WARN_ON (pmd_trans_huge (pmd_set)); + swp_convert =pmd_to_swp_entry (pmd_set); +WARN_ON (swp_type (swp_set) !=swp_type (swp_convert));
mm/debug_vm_pgtable.c static void __init hugetlb_basic_tests(struct pgtable_debug_args *args) { } · line 996
Quoted 10 lines
mm/debug_vm_pgtable.c +10
-0+WARN_ON (swp_offset (swp_set) !=swp_offset (swp_convert)); +WARN_ON (pmd_present (pmd_clear)); +WARN_ON (pmd_trans_huge (pmd_clear)); + swp_convert =pmd_to_swp_entry (pmd_clear); +WARN_ON (swp_type (swp_clear) !=swp_type (swp_convert)); +WARN_ON (swp_offset (swp_clear) !=swp_offset (swp_convert)); + +/* Now invalidate the pmd. */ + pmd_set =pmd_mkinvalid (pmd_set); + pmd_clear =pmd_mkinvalid (pmd_clear);
[ 0.704452] debug_vm_pgtable: [swp_pmd_mkinvalid_tests ]: invalid: pmd_set=f800000000000000, pmd_clear=7ffffffffe00e00
mm/debug_vm_pgtable.c static void __init hugetlb_basic_tests(struct pgtable_debug_args *args) { } · line 1005
Quoted 9 lines
mm/debug_vm_pgtable.c +9
-0+ +/* +* Since its a swap pmd, invalidation should effectively be a noop and +* the checks we already did should give the same answer. Check the +* invalidation didn't corrupt any fields. +*/ +WARN_ON (pmd_present (pmd_set)); +WARN_ON (pmd_trans_huge (pmd_set)); + swp_convert =pmd_to_swp_entry (pmd_set);
[ 0.706461] debug_vm_pgtable: [swp_pmd_mkinvalid_tests ]: set: swp=7c03ffffffffffff (1f, 3ffffffffffff), convert=7c03ffffffffffff (1f, 3ffffffffffff)
mm/debug_vm_pgtable.c static void __init hugetlb_basic_tests(struct pgtable_debug_args *args) { } · line 990
Quoted 5 lines
mm/debug_vm_pgtable.c +5
-0+WARN_ON (swp_type (swp_set) !=swp_type (swp_convert)); +WARN_ON (swp_offset (swp_set) !=swp_offset (swp_convert)); +WARN_ON (pmd_present (pmd_clear)); +WARN_ON (pmd_trans_huge (pmd_clear)); + swp_convert =pmd_to_swp_entry (pmd_clear);
[ 0.708841] debug_vm_pgtable: [swp_pmd_mkinvalid_tests ]: clear: swp=0 (0, 0), convert=ff8 (0, ff8)
mm/debug_vm_pgtable.c static void __init hugetlb_basic_tests(struct pgtable_debug_args *args) { } · line 1012
Quoted 2 lines
mm/debug_vm_pgtable.c +2
-0+WARN_ON (swp_type (swp_clear) !=swp_type (swp_convert)); +WARN_ON (swp_offset (swp_clear) !=swp_offset (swp_convert));
This line fails on x86_64.
The logs show that the offset is indeed being corrupted by pmd_mkinvalid(); 0 -> 0xff8.
I think this is due to x86's pmd_mkinvalid() assuming the pmd is present; pmd_flags() and pmd_pfn() do all sorts of weird and wonderful things.
So does this take us full circle? Are we now back to modifying the core-mm to never call pmd_mkinvalid() on a non-present entry? If so, then I guess we should remove the arm64 fix from for-next/fixes.
Quoted 20 lines
mm/debug_vm_pgtable.c +7
-1+} +#else +static void __init swp_pmd_mkinvalid_tests (struct pgtable_debug_args *args) { } +#endif /* !__HAVE_ARCH_PMDP_INVALIDATE && CONFIG_ARCH_ENABLE_THP_MIGRATION */ +static void __init pmd_thp_tests (struct pgtable_debug_args *args) {pmd_t pmd;@@ -982,6 +1041,8 @@ static void __init pmd_thp_tests(struct pgtable_debug_args *args)WARN_ON (!pmd_trans_huge (pmd_mkinvalid (pmd_mkhuge (pmd))));WARN_ON (!pmd_present (pmd_mkinvalid (pmd_mkhuge (pmd))));#endif /* __HAVE_ARCH_PMDP_INVALIDATE */ + +swp_pmd_mkinvalid_tests (args); }#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD --2.25.1
Pulling in David, who may be able to advise...
Quoted 191 lines · Ryan Roberts
Zi Yan, I'm hoping you might have some input on the below...
Quoted 13 lines · Ryan Roberts
__split_huge_pmd_locked()can be called for a present THP, devmap or (non-present) migration entry. It callspmdp_invalidate()unconditionally on the pmdp and only determines if it is present or not based on the returned old pmd.But arm64's
pmd_mkinvalid(), called bypmdp_invalidate(), unconditionally sets thePMD_PRESENT_INVALIDflag, which causes futurepmd_present()calls to return true - even for a swap pmd. Therefore any lockless pgtable walker could see the migration entry pmd in this state and start interpretting the fields (e.g.pmd_pfn()) as if it were present, leading to BadThings (TM). GUP-fast appears to be one such lockless pgtable walker.Quoted 4 lines
While the obvious fix is for core-mm to avoid such calls for non-present pmds (
pmdp_invalidate()will also issue TLBI which is not necessary for this case either), all other arches that implementpmd_mkinvalid()do it in such a way that it is robust to being called with a non-present pmd.OK the plot thickens; The tests I wrote to check that
pmd_mkinvalid()is safe for swap entries fails onx86_64. See below...Quoted 65 lines
So it is simpler and safer to make arm64 robust too. This approach means we can even add tests to
debug_vm_pgtable.cto validate the required behaviour.This is a theoretical bug found during code review. I don't have any test case to trigger it in practice.
Fixes 53fa117bb33c ("arm64/mm: Enable THP migration")Signed-off-by Ryan Roberts <[email protected]>---
Hi all,
v1 of this fix [1] took the approach of fixing core-mm to never call
pmdp_invalidate()on a non-present pmd. But Zi Yan highlighted that only arm64 suffers this problem; all other arches are robust. So his suggestion was to instead make arm64 robust in the same way and add tests to validate it. Despite my stated reservations in the context of the v1 discussion, having thought on it for a bit, I now agree with Zi Yan. Hence this post.Andrew has v1 in mm-unstable at the moment, so probably the best thing to do is remove it from there and have this go in through the arm64 tree? Assuming there is agreement that this approach is right one.
This applies on top of v6.9-rc5. Passes all the mm selftests on arm64.
[1] https://lore.kernel.org/linux-mm/[email protected]/
Thanks,
Ryanarch/arm64/include/asm/pgtable.h | 12 +++++-- mm/debug_vm_pgtable.c | 61 ++++++++++++++++++++++++++++++++ 2 files changed, 71 insertions(+), 2 deletions(-)arch/arm64/include/asm/pgtable.h +10
-2@@ -511,8 +511,16 @@ static inline int pmd_trans_huge(pmd_t pmd)static inline pmd_t pmd_mkinvalid (pmd_t pmd) { - pmd =set_pmd_bit (pmd,__pgprot (PMD_PRESENT_INVALID )); - pmd =clear_pmd_bit (pmd,__pgprot (PMD_SECT_VALID )); +/* +* If not valid then either we are already present-invalid or we are +* not-present (i.e. none or swap entry). We must not convert +* not-present to present-invalid. Unbelievably, the core-mm may call +* pmd_mkinvalid() for a swap entry and all other arches can handle it. +*/ +if (pmd_valid (pmd)) { + pmd =set_pmd_bit (pmd,__pgprot (PMD_PRESENT_INVALID )); + pmd =clear_pmd_bit (pmd,__pgprot (PMD_SECT_VALID )); + }return pmd; }mm/debug_vm_pgtable.c +0
-0mm/debug_vm_pgtable.c static void __init hugetlb_basic_tests(struct pgtable_debug_args *args) { } · line 961
Quoted 6 lines
mm/debug_vm_pgtable.c +3
-0@@ -956,6 +956,65 @@ static void __init hugetlb_basic_tests(struct pgtable_debug_args *args) { }#endif /* CONFIG_HUGETLB_PAGE */ #ifdef CONFIG_TRANSPARENT_HUGEPAGE +#if !defined(__HAVE_ARCH_PMDP_INVALIDATE) && defined(CONFIG_ARCH_ENABLE_THP_MIGRATION) +static void __init swp_pmd_mkinvalid_tests(struct pgtable_debug_args *args) +{Printing various values at different locations in this function for debug:
Quoted 6 lines
mm/debug_vm_pgtable.c +6
-0+unsigned long max_swap_offset; +swp_entry_t swp_set, swp_clear, swp_convert; +pmd_t pmd_set, pmd_clear; + +/* +* See generic_max_swapfile_size(): probe the maximum offset, then mm/debug_vm_pgtable.c static void __init hugetlb_basic_tests(struct pgtable_debug_args *args) { } · line 977
Quoted 10 lines
mm/debug_vm_pgtable.c +10
-0+* create swap entry will all possible bits set and a swap entry will +* all bits clear. +*/ + max_swap_offset =swp_offset (pmd_to_swp_entry (swp_entry_to_pmd (swp_entry (0 , ~0UL )))); + swp_set =swp_entry ((1 <<MAX_SWAPFILES_SHIFT ) -1 , max_swap_offset); + swp_clear =swp_entry (0 ,0 ); + +/* Convert to pmd. */ + pmd_set =swp_entry_to_pmd (swp_set); + pmd_clear =swp_entry_to_pmd (swp_clear);[ 0.702163] debug_vm_pgtable: [swp_pmd_mkinvalid_tests ]: valid: pmd_set=f800000000000000, pmd_clear=7fffffffffffe00Quoted 9 lines
mm/debug_vm_pgtable.c +9
-0+ +/* +* Sanity check that the pmds are not-present, not-huge and swap entry +* is recoverable without corruption. +*/ +WARN_ON (pmd_present (pmd_set)); +WARN_ON (pmd_trans_huge (pmd_set)); + swp_convert =pmd_to_swp_entry (pmd_set); +WARN_ON (swp_type (swp_set) !=swp_type (swp_convert));mm/debug_vm_pgtable.c static void __init hugetlb_basic_tests(struct pgtable_debug_args *args) { } · line 996
Quoted 10 lines
mm/debug_vm_pgtable.c +10
-0+WARN_ON (swp_offset (swp_set) !=swp_offset (swp_convert)); +WARN_ON (pmd_present (pmd_clear)); +WARN_ON (pmd_trans_huge (pmd_clear)); + swp_convert =pmd_to_swp_entry (pmd_clear); +WARN_ON (swp_type (swp_clear) !=swp_type (swp_convert)); +WARN_ON (swp_offset (swp_clear) !=swp_offset (swp_convert)); + +/* Now invalidate the pmd. */ + pmd_set =pmd_mkinvalid (pmd_set); + pmd_clear =pmd_mkinvalid (pmd_clear);[ 0.704452] debug_vm_pgtable: [swp_pmd_mkinvalid_tests ]: invalid: pmd_set=f800000000000000, pmd_clear=7ffffffffe00e00mm/debug_vm_pgtable.c static void __init hugetlb_basic_tests(struct pgtable_debug_args *args) { } · line 1005
Quoted 9 lines
mm/debug_vm_pgtable.c +9
-0+ +/* +* Since its a swap pmd, invalidation should effectively be a noop and +* the checks we already did should give the same answer. Check the +* invalidation didn't corrupt any fields. +*/ +WARN_ON (pmd_present (pmd_set)); +WARN_ON (pmd_trans_huge (pmd_set)); + swp_convert =pmd_to_swp_entry (pmd_set);[ 0.706461] debug_vm_pgtable: [swp_pmd_mkinvalid_tests ]: set: swp=7c03ffffffffffff (1f, 3ffffffffffff), convert=7c03ffffffffffff (1f, 3ffffffffffff)mm/debug_vm_pgtable.c static void __init hugetlb_basic_tests(struct pgtable_debug_args *args) { } · line 990
Quoted 5 lines
mm/debug_vm_pgtable.c +5
-0+WARN_ON (swp_type (swp_set) !=swp_type (swp_convert)); +WARN_ON (swp_offset (swp_set) !=swp_offset (swp_convert)); +WARN_ON (pmd_present (pmd_clear)); +WARN_ON (pmd_trans_huge (pmd_clear)); + swp_convert =pmd_to_swp_entry (pmd_clear);[ 0.708841] debug_vm_pgtable: [swp_pmd_mkinvalid_tests ]: clear: swp=0 (0, 0), convert=ff8 (0, ff8)mm/debug_vm_pgtable.c static void __init hugetlb_basic_tests(struct pgtable_debug_args *args) { } · line 1012
Quoted 2 lines
mm/debug_vm_pgtable.c +2
-0+WARN_ON (swp_type (swp_clear) !=swp_type (swp_convert)); +WARN_ON (swp_offset (swp_clear) !=swp_offset (swp_convert));This line fails on
x86_64.The logs show that the offset is indeed being corrupted by
pmd_mkinvalid(); 0 ->0xff8.I think this is due to x86's
pmd_mkinvalid()assuming the pmd is present;pmd_flags()andpmd_pfn()do all sorts of weird and wonderful things.So does this take us full circle? Are we now back to modifying the core-mm to never call
pmd_mkinvalid()on a non-present entry? If so, then I guess we should remove the arm64 fix from for-next/fixes.Quoted 20 lines
mm/debug_vm_pgtable.c +7
-1+} +#else +static void __init swp_pmd_mkinvalid_tests (struct pgtable_debug_args *args) { } +#endif /* !__HAVE_ARCH_PMDP_INVALIDATE && CONFIG_ARCH_ENABLE_THP_MIGRATION */ +static void __init pmd_thp_tests (struct pgtable_debug_args *args) {pmd_t pmd;@@ -982,6 +1041,8 @@ static void __init pmd_thp_tests(struct pgtable_debug_args *args)WARN_ON (!pmd_trans_huge (pmd_mkinvalid (pmd_mkhuge (pmd))));WARN_ON (!pmd_present (pmd_mkinvalid (pmd_mkhuge (pmd))));#endif /* __HAVE_ARCH_PMDP_INVALIDATE */ + +swp_pmd_mkinvalid_tests (args); }#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD --2.25.1
Quoted 173 lines · Ryan Roberts
Pulling in David, who may be able to advise...
Quoted 169 lines · Ryan Roberts
Zi Yan, I'm hoping you might have some input on the below...
Quoted 13 lines · Ryan Roberts
__split_huge_pmd_locked()can be called for a present THP, devmap or (non-present) migration entry. It callspmdp_invalidate()unconditionally on the pmdp and only determines if it is present or not based on the returned old pmd.But arm64's
pmd_mkinvalid(), called bypmdp_invalidate(), unconditionally sets thePMD_PRESENT_INVALIDflag, which causes futurepmd_present()calls to return true - even for a swap pmd. Therefore any lockless pgtable walker could see the migration entry pmd in this state and start interpretting the fields (e.g.pmd_pfn()) as if it were present, leading to BadThings (TM). GUP-fast appears to be one such lockless pgtable walker.Quoted 4 lines
While the obvious fix is for core-mm to avoid such calls for non-present pmds (
pmdp_invalidate()will also issue TLBI which is not necessary for this case either), all other arches that implementpmd_mkinvalid()do it in such a way that it is robust to being called with a non-present pmd.OK the plot thickens; The tests I wrote to check that
pmd_mkinvalid()is safe for swap entries fails onx86_64. See below...Quoted 65 lines
So it is simpler and safer to make arm64 robust too. This approach means we can even add tests to
debug_vm_pgtable.cto validate the required behaviour.This is a theoretical bug found during code review. I don't have any test case to trigger it in practice.
Fixes 53fa117bb33c ("arm64/mm: Enable THP migration")Signed-off-by Ryan Roberts <[email protected]>---
Hi all,
v1 of this fix [1] took the approach of fixing core-mm to never call
pmdp_invalidate()on a non-present pmd. But Zi Yan highlighted that only arm64 suffers this problem; all other arches are robust. So his suggestion was to instead make arm64 robust in the same way and add tests to validate it. Despite my stated reservations in the context of the v1 discussion, having thought on it for a bit, I now agree with Zi Yan. Hence this post.Andrew has v1 in mm-unstable at the moment, so probably the best thing to do is remove it from there and have this go in through the arm64 tree? Assuming there is agreement that this approach is right one.
This applies on top of v6.9-rc5. Passes all the mm selftests on arm64.
[1] https://lore.kernel.org/linux-mm/[email protected]/
Thanks,
Ryanarch/arm64/include/asm/pgtable.h | 12 +++++-- mm/debug_vm_pgtable.c | 61 ++++++++++++++++++++++++++++++++ 2 files changed, 71 insertions(+), 2 deletions(-)arch/arm64/include/asm/pgtable.h +10
-2@@ -511,8 +511,16 @@ static inline int pmd_trans_huge(pmd_t pmd)static inline pmd_t pmd_mkinvalid (pmd_t pmd) { - pmd =set_pmd_bit (pmd,__pgprot (PMD_PRESENT_INVALID )); - pmd =clear_pmd_bit (pmd,__pgprot (PMD_SECT_VALID )); +/* +* If not valid then either we are already present-invalid or we are +* not-present (i.e. none or swap entry). We must not convert +* not-present to present-invalid. Unbelievably, the core-mm may call +* pmd_mkinvalid() for a swap entry and all other arches can handle it. +*/ +if (pmd_valid (pmd)) { + pmd =set_pmd_bit (pmd,__pgprot (PMD_PRESENT_INVALID )); + pmd =clear_pmd_bit (pmd,__pgprot (PMD_SECT_VALID )); + }return pmd; }mm/debug_vm_pgtable.c +0
-0mm/debug_vm_pgtable.c static void __init hugetlb_basic_tests(struct pgtable_debug_args *args) { } · line 961
Quoted 6 lines
mm/debug_vm_pgtable.c +3
-0@@ -956,6 +956,65 @@ static void __init hugetlb_basic_tests(struct pgtable_debug_args *args) { }#endif /* CONFIG_HUGETLB_PAGE */ #ifdef CONFIG_TRANSPARENT_HUGEPAGE +#if !defined(__HAVE_ARCH_PMDP_INVALIDATE) && defined(CONFIG_ARCH_ENABLE_THP_MIGRATION) +static void __init swp_pmd_mkinvalid_tests(struct pgtable_debug_args *args) +{Printing various values at different locations in this function for debug:
Quoted 6 lines
mm/debug_vm_pgtable.c +6
-0+unsigned long max_swap_offset; +swp_entry_t swp_set, swp_clear, swp_convert; +pmd_t pmd_set, pmd_clear; + +/* +* See generic_max_swapfile_size(): probe the maximum offset, then mm/debug_vm_pgtable.c static void __init hugetlb_basic_tests(struct pgtable_debug_args *args) { } · line 977
Quoted 10 lines
mm/debug_vm_pgtable.c +10
-0+* create swap entry will all possible bits set and a swap entry will +* all bits clear. +*/ + max_swap_offset =swp_offset (pmd_to_swp_entry (swp_entry_to_pmd (swp_entry (0 , ~0UL )))); + swp_set =swp_entry ((1 <<MAX_SWAPFILES_SHIFT ) -1 , max_swap_offset); + swp_clear =swp_entry (0 ,0 ); + +/* Convert to pmd. */ + pmd_set =swp_entry_to_pmd (swp_set); + pmd_clear =swp_entry_to_pmd (swp_clear);[ 0.702163] debug_vm_pgtable: [swp_pmd_mkinvalid_tests ]: valid: pmd_set=f800000000000000, pmd_clear=7fffffffffffe00Quoted 9 lines
mm/debug_vm_pgtable.c +9
-0+ +/* +* Sanity check that the pmds are not-present, not-huge and swap entry +* is recoverable without corruption. +*/ +WARN_ON (pmd_present (pmd_set)); +WARN_ON (pmd_trans_huge (pmd_set)); + swp_convert =pmd_to_swp_entry (pmd_set); +WARN_ON (swp_type (swp_set) !=swp_type (swp_convert));mm/debug_vm_pgtable.c static void __init hugetlb_basic_tests(struct pgtable_debug_args *args) { } · line 996
Quoted 10 lines
mm/debug_vm_pgtable.c +10
-0+WARN_ON (swp_offset (swp_set) !=swp_offset (swp_convert)); +WARN_ON (pmd_present (pmd_clear)); +WARN_ON (pmd_trans_huge (pmd_clear)); + swp_convert =pmd_to_swp_entry (pmd_clear); +WARN_ON (swp_type (swp_clear) !=swp_type (swp_convert)); +WARN_ON (swp_offset (swp_clear) !=swp_offset (swp_convert)); + +/* Now invalidate the pmd. */ + pmd_set =pmd_mkinvalid (pmd_set); + pmd_clear =pmd_mkinvalid (pmd_clear);[ 0.704452] debug_vm_pgtable: [swp_pmd_mkinvalid_tests ]: invalid: pmd_set=f800000000000000, pmd_clear=7ffffffffe00e00mm/debug_vm_pgtable.c static void __init hugetlb_basic_tests(struct pgtable_debug_args *args) { } · line 1005
Quoted 9 lines
mm/debug_vm_pgtable.c +9
-0+ +/* +* Since its a swap pmd, invalidation should effectively be a noop and +* the checks we already did should give the same answer. Check the +* invalidation didn't corrupt any fields. +*/ +WARN_ON (pmd_present (pmd_set)); +WARN_ON (pmd_trans_huge (pmd_set)); + swp_convert =pmd_to_swp_entry (pmd_set);[ 0.706461] debug_vm_pgtable: [swp_pmd_mkinvalid_tests ]: set: swp=7c03ffffffffffff (1f, 3ffffffffffff), convert=7c03ffffffffffff (1f, 3ffffffffffff)mm/debug_vm_pgtable.c static void __init hugetlb_basic_tests(struct pgtable_debug_args *args) { } · line 990
Quoted 5 lines
mm/debug_vm_pgtable.c +5
-0+WARN_ON (swp_type (swp_set) !=swp_type (swp_convert)); +WARN_ON (swp_offset (swp_set) !=swp_offset (swp_convert)); +WARN_ON (pmd_present (pmd_clear)); +WARN_ON (pmd_trans_huge (pmd_clear)); + swp_convert =pmd_to_swp_entry (pmd_clear);[ 0.708841] debug_vm_pgtable: [swp_pmd_mkinvalid_tests ]: clear: swp=0 (0, 0), convert=ff8 (0, ff8)mm/debug_vm_pgtable.c static void __init hugetlb_basic_tests(struct pgtable_debug_args *args) { } · line 1012
Quoted 2 lines
mm/debug_vm_pgtable.c +2
-0+WARN_ON (swp_type (swp_clear) !=swp_type (swp_convert)); +WARN_ON (swp_offset (swp_clear) !=swp_offset (swp_convert));This line fails on
x86_64.The logs show that the offset is indeed being corrupted by
pmd_mkinvalid(); 0 ->0xff8.I think this is due to x86's
pmd_mkinvalid()assuming the pmd is present;pmd_flags()andpmd_pfn()do all sorts of weird and wonderful things.So does this take us full circle? Are we now back to modifying the core-mm to never call
pmd_mkinvalid()on a non-present entry? If so, then I guess we should remove the arm64 fix from for-next/fixes.
If x86_64's pmd_mkinvalid() also corrupts swap entries, yes, your original fix is better. I will dig into the x86 code more to figure out what goes wrong. Last time, I only checked PAGE_* bits in these pmd|pte_* operations. Sorry for the misinformation.
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
mm/debug_vm_pgtable.c +7
-0+} +#else +static void __init swp_pmd_mkinvalid_tests (struct pgtable_debug_args *args) { } +#endif /* !__HAVE_ARCH_PMDP_INVALIDATE && CONFIG_ARCH_ENABLE_THP_MIGRATION */ +static void __init pmd_thp_tests (struct pgtable_debug_args *args) {pmd_t pmd;@@ -982,6 +1041,8 @@ static void __init pmd_thp_tests(struct pgtable_debug_args *args)WARN_ON (!pmd_trans_huge (pmd_mkinvalid (pmd_mkhuge (pmd))));WARN_ON (!pmd_present (pmd_mkinvalid (pmd_mkhuge (pmd))));#endif /* __HAVE_ARCH_PMDP_INVALIDATE */ + +swp_pmd_mkinvalid_tests (args); }
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
mm/debug_vm_pgtable.c +0
-1#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD --2.25.1
--
Best Regards,
Yan, Zi
Quoted 172 lines · Zi Yan
Quoted 170 lines · Ryan Roberts
Pulling in David, who may be able to advise...
Quoted 166 lines · Ryan Roberts
Zi Yan, I'm hoping you might have some input on the below...
Quoted 13 lines · Ryan Roberts
__split_huge_pmd_locked()can be called for a present THP, devmap or (non-present) migration entry. It callspmdp_invalidate()unconditionally on the pmdp and only determines if it is present or not based on the returned old pmd.But arm64's
pmd_mkinvalid(), called bypmdp_invalidate(), unconditionally sets thePMD_PRESENT_INVALIDflag, which causes futurepmd_present()calls to return true - even for a swap pmd. Therefore any lockless pgtable walker could see the migration entry pmd in this state and start interpretting the fields (e.g.pmd_pfn()) as if it were present, leading to BadThings (TM). GUP-fast appears to be one such lockless pgtable walker.Quoted 4 lines
While the obvious fix is for core-mm to avoid such calls for non-present pmds (
pmdp_invalidate()will also issue TLBI which is not necessary for this case either), all other arches that implementpmd_mkinvalid()do it in such a way that it is robust to being called with a non-present pmd.OK the plot thickens; The tests I wrote to check that
pmd_mkinvalid()is safe for swap entries fails onx86_64. See below...Quoted 65 lines
So it is simpler and safer to make arm64 robust too. This approach means we can even add tests to
debug_vm_pgtable.cto validate the required behaviour.This is a theoretical bug found during code review. I don't have any test case to trigger it in practice.
Fixes 53fa117bb33c ("arm64/mm: Enable THP migration")Signed-off-by Ryan Roberts <[email protected]>---
Hi all,
v1 of this fix [1] took the approach of fixing core-mm to never call
pmdp_invalidate()on a non-present pmd. But Zi Yan highlighted that only arm64 suffers this problem; all other arches are robust. So his suggestion was to instead make arm64 robust in the same way and add tests to validate it. Despite my stated reservations in the context of the v1 discussion, having thought on it for a bit, I now agree with Zi Yan. Hence this post.Andrew has v1 in mm-unstable at the moment, so probably the best thing to do is remove it from there and have this go in through the arm64 tree? Assuming there is agreement that this approach is right one.
This applies on top of v6.9-rc5. Passes all the mm selftests on arm64.
[1] https://lore.kernel.org/linux-mm/[email protected]/
Thanks,
Ryanarch/arm64/include/asm/pgtable.h | 12 +++++-- mm/debug_vm_pgtable.c | 61 ++++++++++++++++++++++++++++++++ 2 files changed, 71 insertions(+), 2 deletions(-)arch/arm64/include/asm/pgtable.h +10
-2@@ -511,8 +511,16 @@ static inline int pmd_trans_huge(pmd_t pmd)static inline pmd_t pmd_mkinvalid (pmd_t pmd) { - pmd =set_pmd_bit (pmd,__pgprot (PMD_PRESENT_INVALID )); - pmd =clear_pmd_bit (pmd,__pgprot (PMD_SECT_VALID )); +/* +* If not valid then either we are already present-invalid or we are +* not-present (i.e. none or swap entry). We must not convert +* not-present to present-invalid. Unbelievably, the core-mm may call +* pmd_mkinvalid() for a swap entry and all other arches can handle it. +*/ +if (pmd_valid (pmd)) { + pmd =set_pmd_bit (pmd,__pgprot (PMD_PRESENT_INVALID )); + pmd =clear_pmd_bit (pmd,__pgprot (PMD_SECT_VALID )); + }return pmd; }mm/debug_vm_pgtable.c +0
-0mm/debug_vm_pgtable.c static void __init hugetlb_basic_tests(struct pgtable_debug_args *args) { } · line 961
Quoted 6 lines
mm/debug_vm_pgtable.c +3
-0@@ -956,6 +956,65 @@ static void __init hugetlb_basic_tests(struct pgtable_debug_args *args) { }#endif /* CONFIG_HUGETLB_PAGE */ #ifdef CONFIG_TRANSPARENT_HUGEPAGE +#if !defined(__HAVE_ARCH_PMDP_INVALIDATE) && defined(CONFIG_ARCH_ENABLE_THP_MIGRATION) +static void __init swp_pmd_mkinvalid_tests(struct pgtable_debug_args *args) +{Printing various values at different locations in this function for debug:
Quoted 6 lines
mm/debug_vm_pgtable.c +6
-0+unsigned long max_swap_offset; +swp_entry_t swp_set, swp_clear, swp_convert; +pmd_t pmd_set, pmd_clear; + +/* +* See generic_max_swapfile_size(): probe the maximum offset, then mm/debug_vm_pgtable.c static void __init hugetlb_basic_tests(struct pgtable_debug_args *args) { } · line 977
Quoted 10 lines
mm/debug_vm_pgtable.c +10
-0+* create swap entry will all possible bits set and a swap entry will +* all bits clear. +*/ + max_swap_offset =swp_offset (pmd_to_swp_entry (swp_entry_to_pmd (swp_entry (0 , ~0UL )))); + swp_set =swp_entry ((1 <<MAX_SWAPFILES_SHIFT ) -1 , max_swap_offset); + swp_clear =swp_entry (0 ,0 ); + +/* Convert to pmd. */ + pmd_set =swp_entry_to_pmd (swp_set); + pmd_clear =swp_entry_to_pmd (swp_clear);[ 0.702163] debug_vm_pgtable: [swp_pmd_mkinvalid_tests ]: valid: pmd_set=f800000000000000, pmd_clear=7fffffffffffe00Quoted 9 lines
mm/debug_vm_pgtable.c +9
-0+ +/* +* Sanity check that the pmds are not-present, not-huge and swap entry +* is recoverable without corruption. +*/ +WARN_ON (pmd_present (pmd_set)); +WARN_ON (pmd_trans_huge (pmd_set)); + swp_convert =pmd_to_swp_entry (pmd_set); +WARN_ON (swp_type (swp_set) !=swp_type (swp_convert));mm/debug_vm_pgtable.c static void __init hugetlb_basic_tests(struct pgtable_debug_args *args) { } · line 996
Quoted 10 lines
mm/debug_vm_pgtable.c +10
-0+WARN_ON (swp_offset (swp_set) !=swp_offset (swp_convert)); +WARN_ON (pmd_present (pmd_clear)); +WARN_ON (pmd_trans_huge (pmd_clear)); + swp_convert =pmd_to_swp_entry (pmd_clear); +WARN_ON (swp_type (swp_clear) !=swp_type (swp_convert)); +WARN_ON (swp_offset (swp_clear) !=swp_offset (swp_convert)); + +/* Now invalidate the pmd. */ + pmd_set =pmd_mkinvalid (pmd_set); + pmd_clear =pmd_mkinvalid (pmd_clear);[ 0.704452] debug_vm_pgtable: [swp_pmd_mkinvalid_tests ]: invalid: pmd_set=f800000000000000, pmd_clear=7ffffffffe00e00mm/debug_vm_pgtable.c static void __init hugetlb_basic_tests(struct pgtable_debug_args *args) { } · line 1005
Quoted 9 lines
mm/debug_vm_pgtable.c +9
-0+ +/* +* Since its a swap pmd, invalidation should effectively be a noop and +* the checks we already did should give the same answer. Check the +* invalidation didn't corrupt any fields. +*/ +WARN_ON (pmd_present (pmd_set)); +WARN_ON (pmd_trans_huge (pmd_set)); + swp_convert =pmd_to_swp_entry (pmd_set);[ 0.706461] debug_vm_pgtable: [swp_pmd_mkinvalid_tests ]: set: swp=7c03ffffffffffff (1f, 3ffffffffffff), convert=7c03ffffffffffff (1f, 3ffffffffffff)mm/debug_vm_pgtable.c static void __init hugetlb_basic_tests(struct pgtable_debug_args *args) { } · line 990
Quoted 5 lines
mm/debug_vm_pgtable.c +5
-0+WARN_ON (swp_type (swp_set) !=swp_type (swp_convert)); +WARN_ON (swp_offset (swp_set) !=swp_offset (swp_convert)); +WARN_ON (pmd_present (pmd_clear)); +WARN_ON (pmd_trans_huge (pmd_clear)); + swp_convert =pmd_to_swp_entry (pmd_clear);[ 0.708841] debug_vm_pgtable: [swp_pmd_mkinvalid_tests ]: clear: swp=0 (0, 0), convert=ff8 (0, ff8)mm/debug_vm_pgtable.c static void __init hugetlb_basic_tests(struct pgtable_debug_args *args) { } · line 1012
Quoted 2 lines
mm/debug_vm_pgtable.c +2
-0+WARN_ON (swp_type (swp_clear) !=swp_type (swp_convert)); +WARN_ON (swp_offset (swp_clear) !=swp_offset (swp_convert));This line fails on
x86_64.The logs show that the offset is indeed being corrupted by
pmd_mkinvalid(); 0 ->0xff8.
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
I think this is due to x86's
pmd_mkinvalid()assuming the pmd is present;pmd_flags()andpmd_pfn()do all sorts of weird and wonderful things.So does this take us full circle? Are we now back to modifying the core-mm to never call
pmd_mkinvalid()on a non-present entry? If so, then I guess we should remove the arm64 fix from for-next/fixes.If x86_64's
pmd_mkinvalid()also corrupts swap entries, yes, your original fix is better. I will dig into the x86 code more to figure out what goes wrong. Last time, I only checked PAGE_* bits in these pmd|pte_* operations. Sorry for the misinformation.
No worries, I'll do the amends we originally agreed for the original fix and resend.
Quoted 28 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
mm/debug_vm_pgtable.c +7
-0+} +#else +static void __init swp_pmd_mkinvalid_tests (struct pgtable_debug_args *args) { } +#endif /* !__HAVE_ARCH_PMDP_INVALIDATE && CONFIG_ARCH_ENABLE_THP_MIGRATION */ +static void __init pmd_thp_tests (struct pgtable_debug_args *args) {pmd_t pmd;@@ -982,6 +1041,8 @@ static void __init pmd_thp_tests(struct pgtable_debug_args *args)WARN_ON (!pmd_trans_huge (pmd_mkinvalid (pmd_mkhuge (pmd))));WARN_ON (!pmd_present (pmd_mkinvalid (pmd_mkhuge (pmd))));#endif /* __HAVE_ARCH_PMDP_INVALIDATE */ + +swp_pmd_mkinvalid_tests (args); }Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
mm/debug_vm_pgtable.c +0
-1#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD --2.25.1
--
Best Regards,
Yan, Zi