arm64/mm: Fix use-after-free due to race between memory hotunplug and ptdump
Subsystems (2)
PATCH 1/1 arm64/mm: Fix use-after-free due to race between memory hotunplug and ptdump
Memory hotunplug is done under the hotplug lock and ptdump walk is done under the init_mm.mmap_lock. Therefore, ptdump and hotunplug can run simultaneously without any synchronization. During hotunplug, free_empty_tables() is ultimately called to free up the pagetables. The following race can happen, where x denotes the level of the pagetable:
CPU1 CPU2free_empty_pxd_table
ptdump_walk_pgd() Get p(x+1)d table from pxd entrypxd_clear
free_hotplug_pgtable_page(p(x+1)dp)
Still using the p(x+1)d table
which leads to a user-after-free.
To solve this, we need to synchronize ptdump_walk_pgd() with free_hotplug_pgtable_page() in such a way that ptdump never takes a reference on a freed pagetable.
Since this race is very unlikely to happen in practice, we do not want to penalize other code paths taking the init_mm mmap_lock. Therefore, we use static keys. ptdump will enable the static key - upon observing that, the free_empty_pxd_table() functions will get patched in with an mmap_read_lock/unlock sequence. A code comment explains in detail, how a combination of acquire semantics of static_branch_enable() and the barriers in __flush_tlb_kernel_pgtable() ensures that ptdump will never get a hold on the address of a freed pagetable - either ptdump will block the table freeing path due to write locking the mmap_lock, or, the nullity of the pxd entry will be observed by ptdump, therefore having no access to the isolated p(x+1)d pagetable.
This bug was found by code inspection, as a result of working on [1].
1. https://lore.kernel.org/all/[email protected]/
---
Rebased on Linux 6.16.
arch/arm64/include/asm/ptdump.h | 2 ++ arch/arm64/mm/mmu.c | 61 +++++++++++++++++++++++++++++++++ arch/arm64/mm/ptdump.c | 11 ++++-- 3 files changed, 72 insertions(+), 2 deletions(-)
arch/arm64/include/asm/ptdump.h +2 -0
@@ -7,6 +7,8 @@#include <linux/ptdump.h> +DECLARE_STATIC_KEY_FALSE (arm64_ptdump_key); +#ifdef CONFIG_PTDUMP #include <linux/mm_types.h>
arch/arm64/mm/mmu.c +61 -0
97 lines not on this page · raw message
arch/arm64/mm/ptdump.c +9 -2
@@ -283,6 +283,13 @@ void note_page_flush(struct ptdump_state *pt_st)note_page (pt_st,0 ,-1 ,pte_val (pte_zero)); } +static void arm64_ptdump_walk_pgd (struct ptdump_state *st,struct mm_struct *mm) +{ +static_branch_enable (&arm64_ptdump_key); +ptdump_walk_pgd (st, mm,NULL ); +static_branch_disable (&arm64_ptdump_key); +} +void ptdump_walk (struct seq_file *s,struct ptdump_info *info) {unsigned long end = ~0UL ;
@@ -311,7 +318,7 @@ void ptdump_walk(struct seq_file *s, struct ptdump_info *info) } }; -ptdump_walk_pgd (&st.ptdump, info->mm,NULL ); +arm64_ptdump_walk_pgd (&st.ptdump, info->mm); }static void __init ptdump_initialize (void )
@@ -353,7 +360,7 @@ bool ptdump_check_wx(void) } }; -ptdump_walk_pgd (&st.ptdump, &init_mm,NULL ); +arm64_ptdump_walk_pgd (&st.ptdump, &init_mm);if (st.wx_pages || st.uxn_pages) { pr_warn("Checked W+X mappings: FAILED, %lu W+X pages found, %lu non-UXN pages found\n",
Signature · 1 line
2.30.2
Quoted 32 lines · Dev Jain
Memory hotunplug is done under the hotplug lock and ptdump walk is done under the init_mm.mmap_lock. Therefore, ptdump and hotunplug can run simultaneously without any synchronization. During hotunplug,
free_empty_tables()is ultimately called to free up the pagetables. The following race can happen, where x denotes the level of the pagetable:CPU1 CPU2
free_empty_pxd_table
ptdump_walk_pgd()Get p(x+1)d table from pxd entrypxd_clear
free_hotplug_pgtable_page(p(x+1)dp)
Still using the p(x+1)d tablewhich leads to a user-after-free.
To solve this, we need to synchronize
ptdump_walk_pgd()withfree_hotplug_pgtable_page()in such a way that ptdump never takes a reference on a freed pagetable.Since this race is very unlikely to happen in practice, we do not want to penalize other code paths taking the
init_mmmmap_lock. Therefore, we use static keys. ptdump will enable the static key - upon observing that, thefree_empty_pxd_table()functions will get patched in with an mmap_read_lock/unlock sequence. A code comment explains in detail, how a combination of acquire semantics ofstatic_branch_enable()and the barriers in__flush_tlb_kernel_pgtable()ensures that ptdump will never get a hold on the address of a freed pagetable - either ptdump will block the table freeing path due to write locking themmap_lock, or, the nullity of the pxd entry will be observed by ptdump, therefore having no access to the isolated p(x+1)d pagetable.
Quoted 7 lines
This bug was found by code inspection, as a result of working on [1].
1. https://lore.kernel.org/all/[email protected]/Cc <[email protected]>Fixes bbd6ec605c0f ("arm64/mm: Enable memory hot remove")Signed-off-by Dev Jain <[email protected]>---
Immediately after posting, I guess the first objection which is going to come is, why not just nest free_empty_tables() with mmap_read_lock/unlock. Memory offlining obviously should not be a hot path so taking the read lock should be fine I guess.
Quoted 14 lines · Dev Jain
Memory hotunplug is done under the hotplug lock and ptdump walk is done under the init_mm.mmap_lock. Therefore, ptdump and hotunplug can run simultaneously without any synchronization. During hotunplug,
free_empty_tables()is ultimately called to free up the pagetables. The following race can happen, where x denotes the level of the pagetable:CPU1 CPU2
free_empty_pxd_table
ptdump_walk_pgd()Get p(x+1)d table from pxd entrypxd_clear
free_hotplug_pgtable_page(p(x+1)dp)
Still using the p(x+1)d table
Quoted 1 line
which leads to a user-after-free.
I'm not sure I understand this. ptdump_show() protects against this with get_online_mems()/put_online_mems(), doesn't it? There are 2 paths that call ptdump_walk_pgd(). This protects one of them. The other is ptdump_check_wx(); I thought you (or Anshuman?) had a patch in flight to fix that with [get|put]_online_mems() too?
Sorry if my memory is failing me here...
Quoted 15 lines · Ryan Roberts
Quoted 14 lines · Dev Jain
Memory hotunplug is done under the hotplug lock and ptdump walk is done under the init_mm.mmap_lock. Therefore, ptdump and hotunplug can run simultaneously without any synchronization. During hotunplug,
free_empty_tables()is ultimately called to free up the pagetables. The following race can happen, where x denotes the level of the pagetable:CPU1 CPU2
free_empty_pxd_table
ptdump_walk_pgd()Get p(x+1)d table from pxd entrypxd_clear
free_hotplug_pgtable_page(p(x+1)dp)
Still using the p(x+1)d table
Quoted 8 lines
Quoted 1 line
which leads to a user-after-free.
I'm not sure I understand this.
ptdump_show()protects against this with get_online_mems()/put_online_mems(), doesn't it? There are 2 paths that callptdump_walk_pgd(). This protects one of them. The other isptdump_check_wx(); I thought you (or Anshuman?) had a patch in flight to fix that with [get|put]_online_mems() too?Sorry if my memory is failing me here...
Nope, I think I just had a use-after-free in my memory so I came up with this patch :) Because of the recent work with ptdump, I was so concentrated on ptdump_walk_pgd() that I didn't even bother looking up the call chain. And I even forgot we had these [get|put]_online_mems() patches recently.
Sorry for the noise, it must have been incredibly confusing to see this patch :(
Quoted 26 lines · Dev Jain
Quoted 15 lines · Ryan Roberts
Quoted 14 lines · Dev Jain
Memory hotunplug is done under the hotplug lock and ptdump walk is done under the init_mm.mmap_lock. Therefore, ptdump and hotunplug can run simultaneously without any synchronization. During hotunplug,
free_empty_tables()is ultimately called to free up the pagetables. The following race can happen, where x denotes the level of the pagetable:CPU1 CPU2
free_empty_pxd_table
ptdump_walk_pgd()
Get p(x+1)d table from pxd entrypxd_clearfree_hotplug_pgtable_page(p(x+1)dp)
Still using the p(x+1)d tableQuoted 8 lines
Quoted 1 line
which leads to a user-after-free.
I'm not sure I understand this.
ptdump_show()protects against this with get_online_mems()/put_online_mems(), doesn't it? There are 2 paths that callptdump_walk_pgd(). This protects one of them. The other isptdump_check_wx(); I thought you (or Anshuman?) had a patch in flight to fix that with [get|put]_online_mems() too?Sorry if my memory is failing me here...
Quoted 7 lines
Nope, I think I just had a use-after-free in my memory so I came up with this patch :)
Because of the recent work with ptdump, I was so concentrated onptdump_walk_pgd()that I
didn't even bother looking up the call chain. And I even forgot we had these [get|put]_online_mems()
patches recently.
I just checked; Anshuman's fix is in mm-stable, so I guess it'll be in v6.17-rc1.
That's the patch:
https://lore.kernel.org/linux-arm-kernel/[email protected]/
Quoted 3 lines
Sorry for the noise, it must have been incredibly confusing to see this patch :(