x86/mm: don't free p4d table when it is folded at runtime.
PATCH 1/1 x86/mm: don't free p4d table when it is folded at runtime.
When the p4d page table layer is folded at runtime, the p4d_free() should do nothing, the same as in <asm-generic/pgtable-nop4d.h>.
It seems this bug should cause double-free in efi_call_phys_epilog(), but I don't know how to trigger that code path, so I can't confirm that by testing.
--- arch/x86/include/asm/pgalloc.h | 3 +++ 1 file changed, 3 insertions(+)
arch/x86/include/asm/pgalloc.h +3 -0
@@ -184,6 +184,9 @@ static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr)static inline void p4d_free (struct mm_struct *mm,p4d_t *p4d) { +if (!pgtable_l5_enabled ()) +return ; +BUG_ON ((unsigned long )p4d & (PAGE_SIZE -1 ));free_page ((unsigned long )p4d); }
Signature · 1 line
2.16.4
Quoted 6 lines · Andrey Ryabinin
When the p4d page table layer is folded at runtime, the
p4d_free()should do nothing, the same as in <asm-generic/pgtable-nop4d.h>.It seems this bug should cause double-free in
efi_call_phys_epilog(), but I don't know how to trigger that code path, so I can't confirm that by testing.
+ Baoquan, Matt.
There's other bug in the efi_call_phys_epilog() that prevents the bug from being triggered.
With the patch below. You can trigger the bug with efi=old_map in kernel command line + KALSR and CONFIG_X86_5LEVEL=y:
page:fffff6bec0000000 count:0 mapcount:1 mapping:0000000000000000 index:0x0 flags: 0x800(reserved) raw: 0000000000000800 fffff6bec0000008 fffff6bec0000008 0000000000000000 raw: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 page dumped because: VM_BUG_ON_PAGE(page_ref_count(page) == 0) ------------[ cut here ]------------ kernel BUG at /home/kas/linux/la57/include/linux/mm.h:499! invalid opcode: 0000 [#1] PREEMPT SMP CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.18.0-rc2-00037-g6f0d349d922b-dirty #58 Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015 RIP: 0010:__free_pages+0x28/0x30 Code: 00 00 8b 47 34 85 c0 74 15 f0 ff 4f 34 75 09 85 f6 74 06 e9 ca d8 ff ff c3 e9 64 ff ff ff 48 RSP: 0000:ffffffff9a403e90 EFLAGS: 00000246 RAX: 000000000000003e RBX: ffffffff9a41d000 RCX: 0000000000000002 RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00000000ffffffff RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000000 R12: ffff9092af089000 R13: ffffffff9a598a80 R14: 0000000000000001 R15: 0000000000000001 FS: 0000000000000000(0000) GS:ffff9092bfc00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ffff9092bffff000 CR3: 0000000198e1d000 CR4: 00000000000006b0 Call Trace: efi_call_phys_epilog+0x17d/0x1bb efi_enter_virtual_mode+0x457/0x4ca start_kernel+0x443/0x4dc secondary_startup_64+0xb7/0xc0 Modules linked in: ---[ end trace 61e271260b11acdd ]---
I'll send patch for efi_call_phys_epilog().
Quoted 3 lines
Fixes 98219dda2ab5 ("x86/mm: Fold p4d page table layer at runtime")Signed-off-by Andrey Ryabinin <[email protected]>
Quoted 9 lines
--- arch/x86/include/asm/pgalloc.h | 3 +++ 1 file changed, 3 insertions(+)arch/x86/include/asm/pgalloc.h +0
-0
arch/x86/include/asm/pgalloc.h static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) · line 192
Quoted 9 lines
arch/x86/include/asm/pgalloc.h +3
-0@@ -184,6 +184,9 @@ static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr)static inline void p4d_free (struct mm_struct *mm,p4d_t *p4d) { +if (!pgtable_l5_enabled ()) +return ; +BUG_ON ((unsigned long )p4d & (PAGE_SIZE -1 ));free_page ((unsigned long )p4d); }
arch/x86/platform/efi/efi_64.c +2 -2
@@ -166,14 +166,14 @@ void __init efi_call_phys_epilog(pgd_t *save_pgd) pgd =pgd_offset_k (pgd_idx *PGDIR_SIZE );set_pgd (pgd_offset_k (pgd_idx *PGDIR_SIZE ), save_pgd[pgd_idx]); -if (!(pgd_val (*pgd) & _PAGE_PRESENT)) +if (!pgd_present (*pgd))continue ;for (i =0 ; i <PTRS_PER_P4D ; i++) { p4d =p4d_offset (pgd, pgd_idx *PGDIR_SIZE + i *P4D_SIZE ); -if (!(p4d_val (*p4d) & _PAGE_PRESENT)) +if (!p4d_present (*p4d))continue ; pud = (pud_t *)p4d_page_vaddr (*p4d);
Signature · 1 line
Kirill A. Shutemov
Hi Kirill,
Quoted 78 lines · Kirill A. Shutemov
Quoted 6 lines · Andrey Ryabinin
When the p4d page table layer is folded at runtime, the
p4d_free()should do nothing, the same as in <asm-generic/pgtable-nop4d.h>.It seems this bug should cause double-free in
efi_call_phys_epilog(), but I don't know how to trigger that code path, so I can't confirm that by testing.+ Baoquan, Matt.
There's other bug in the
efi_call_phys_epilog()that prevents the bug from being triggered.With the patch below. You can trigger the bug with efi=old_map in kernel command line + KALSR and
CONFIG_X86_5LEVEL=y:page:fffff6bec0000000 count:0 mapcount:1 mapping:0000000000000000 index:0x0 flags: 0x800(reserved) raw: 0000000000000800 fffff6bec0000008 fffff6bec0000008 0000000000000000 raw: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 page dumped because: VM_BUG_ON_PAGE(page_ref_count(page) == 0) ------------[ cut here ]------------ kernel BUG at /home/kas/linux/la57/include/linux/mm.h:499! invalid opcode: 0000 [#1] PREEMPT SMP CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.18.0-rc2-00037-g6f0d349d922b-dirty #58 Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015 RIP: 0010:__free_pages+0x28/0x30 Code: 00 00 8b 47 34 85 c0 74 15 f0 ff 4f 34 75 09 85 f6 74 06 e9 ca d8 ff ff c3 e9 64 ff ff ff 48 RSP: 0000:ffffffff9a403e90 EFLAGS: 00000246 RAX: 000000000000003e RBX: ffffffff9a41d000 RCX: 0000000000000002 RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00000000ffffffff RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000000 R12: ffff9092af089000 R13: ffffffff9a598a80 R14: 0000000000000001 R15: 0000000000000001 FS: 0000000000000000(0000) GS:ffff9092bfc00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ffff9092bffff000 CR3: 0000000198e1d000 CR4: 00000000000006b0 Call Trace: efi_call_phys_epilog+0x17d/0x1bb efi_enter_virtual_mode+0x457/0x4ca start_kernel+0x443/0x4dc secondary_startup_64+0xb7/0xc0 Modules linked in: ---[ end trace 61e271260b11acdd ]---I'll send patch for
efi_call_phys_epilog().Quoted 3 lines
Fixes 98219dda2ab5 ("x86/mm: Fold p4d page table layer at runtime")Signed-off-by Andrey Ryabinin <[email protected]>Reviewed-by Kirill A. Shutemov <[email protected]>Cc [email protected] # 4.17Quoted 18 lines
--- arch/x86/include/asm/pgalloc.h | 3 +++ 1 file changed, 3 insertions(+)arch/x86/include/asm/pgalloc.h +3
-0@@ -184,6 +184,9 @@ static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr)static inline void p4d_free (struct mm_struct *mm,p4d_t *p4d) { +if (!pgtable_l5_enabled ()) +return ; +BUG_ON ((unsigned long )p4d & (PAGE_SIZE -1 ));free_page ((unsigned long )p4d); }arch/x86/platform/efi/efi_64.c +0
-0
arch/x86/platform/efi/efi_64.c void __init efi_call_phys_epilog(pgd_t *save_pgd) · line 170
Quoted 6 lines
arch/x86/platform/efi/efi_64.c +1
-1@@ -166,14 +166,14 @@ void __init efi_call_phys_epilog(pgd_t *save_pgd) pgd =pgd_offset_k (pgd_idx *PGDIR_SIZE );set_pgd (pgd_offset_k (pgd_idx *PGDIR_SIZE ), save_pgd[pgd_idx]); -if (!(pgd_val (*pgd) & _PAGE_PRESENT)) +if (!pgd_present (*pgd))continue ;
I may not understand boot-time p4d folding. Here p4d is folded, why pgd_present() need always return 1 if p4d folded?
Thanks
Baoquan
Quoted 12 lines
arch/x86/platform/efi/efi_64.c +1
-2for (i =0 ; i <PTRS_PER_P4D ; i++) { p4d =p4d_offset (pgd, pgd_idx *PGDIR_SIZE + i *P4D_SIZE ); -if (!(p4d_val (*p4d) & _PAGE_PRESENT)) +if (!p4d_present (*p4d))continue ; pud = (pud_t *)p4d_page_vaddr (*p4d); -- Kirill A. Shutemov
Quoted 84 lines · Baoquan He
Hi Kirill,
Quoted 81 lines · Kirill A. Shutemov
Quoted 6 lines · Andrey Ryabinin
When the p4d page table layer is folded at runtime, the
p4d_free()should do nothing, the same as in <asm-generic/pgtable-nop4d.h>.It seems this bug should cause double-free in
efi_call_phys_epilog(), but I don't know how to trigger that code path, so I can't confirm that by testing.+ Baoquan, Matt.
There's other bug in the
efi_call_phys_epilog()that prevents the bug from being triggered.With the patch below. You can trigger the bug with efi=old_map in kernel command line + KALSR and
CONFIG_X86_5LEVEL=y:page:fffff6bec0000000 count:0 mapcount:1 mapping:0000000000000000 index:0x0 flags: 0x800(reserved) raw: 0000000000000800 fffff6bec0000008 fffff6bec0000008 0000000000000000 raw: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 page dumped because: VM_BUG_ON_PAGE(page_ref_count(page) == 0) ------------[ cut here ]------------ kernel BUG at /home/kas/linux/la57/include/linux/mm.h:499! invalid opcode: 0000 [#1] PREEMPT SMP CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.18.0-rc2-00037-g6f0d349d922b-dirty #58 Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015 RIP: 0010:__free_pages+0x28/0x30 Code: 00 00 8b 47 34 85 c0 74 15 f0 ff 4f 34 75 09 85 f6 74 06 e9 ca d8 ff ff c3 e9 64 ff ff ff 48 RSP: 0000:ffffffff9a403e90 EFLAGS: 00000246 RAX: 000000000000003e RBX: ffffffff9a41d000 RCX: 0000000000000002 RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00000000ffffffff RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000000 R12: ffff9092af089000 R13: ffffffff9a598a80 R14: 0000000000000001 R15: 0000000000000001 FS: 0000000000000000(0000) GS:ffff9092bfc00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ffff9092bffff000 CR3: 0000000198e1d000 CR4: 00000000000006b0 Call Trace: efi_call_phys_epilog+0x17d/0x1bb efi_enter_virtual_mode+0x457/0x4ca start_kernel+0x443/0x4dc secondary_startup_64+0xb7/0xc0 Modules linked in: ---[ end trace 61e271260b11acdd ]---I'll send patch for
efi_call_phys_epilog().Quoted 3 lines
Fixes 98219dda2ab5 ("x86/mm: Fold p4d page table layer at runtime")Signed-off-by Andrey Ryabinin <[email protected]>Reviewed-by Kirill A. Shutemov <[email protected]>Cc [email protected] # 4.17Quoted 18 lines
--- arch/x86/include/asm/pgalloc.h | 3 +++ 1 file changed, 3 insertions(+)arch/x86/include/asm/pgalloc.h +3
-0@@ -184,6 +184,9 @@ static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr)static inline void p4d_free (struct mm_struct *mm,p4d_t *p4d) { +if (!pgtable_l5_enabled ()) +return ; +BUG_ON ((unsigned long )p4d & (PAGE_SIZE -1 ));free_page ((unsigned long )p4d); }arch/x86/platform/efi/efi_64.c +0
-0@@ -166,14 +166,14 @@ void __init efi_call_phys_epilog(pgd_t *save_pgd) pgd =pgd_offset_k (pgd_idx *PGDIR_SIZE );set_pgd (pgd_offset_k (pgd_idx *PGDIR_SIZE ), save_pgd[pgd_idx]);
Quoted 6 lines
arch/x86/platform/efi/efi_64.c void __init efi_call_phys_epilog(pgd_t *save_pgd) · line 170
Quoted 3 lines
arch/x86/platform/efi/efi_64.c +1
-1-if (!(pgd_val (*pgd) & _PAGE_PRESENT)) +if (!pgd_present (*pgd))continue ;I may not understand boot-time p4d folding. Here p4d is folded, why
pgd_present()need always return 1 if p4d folded?
Yeah. This is confusing. Basically, we pretend that p4d level consist of 1 entry, with the same value as pgd above. We say that pgd is always present and all checks happens on p4d level.
It's not specific to boot-time folding. Compilet-time folding doing the same. See include/asm-generic/pgtable-nop4d.h (and the rest -nop?d.h)
Signature · 1 line
Kirill A. Shutemov
Quoted 92 lines · Kirill A. Shutemov
Quoted 84 lines · Baoquan He
Hi Kirill,
Quoted 81 lines · Kirill A. Shutemov
Quoted 6 lines · Andrey Ryabinin
When the p4d page table layer is folded at runtime, the
p4d_free()should do nothing, the same as in <asm-generic/pgtable-nop4d.h>.It seems this bug should cause double-free in
efi_call_phys_epilog(), but I don't know how to trigger that code path, so I can't confirm that by testing.+ Baoquan, Matt.
There's other bug in the
efi_call_phys_epilog()that prevents the bug from being triggered.With the patch below. You can trigger the bug with efi=old_map in kernel command line + KALSR and
CONFIG_X86_5LEVEL=y:page:fffff6bec0000000 count:0 mapcount:1 mapping:0000000000000000 index:0x0 flags: 0x800(reserved) raw: 0000000000000800 fffff6bec0000008 fffff6bec0000008 0000000000000000 raw: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 page dumped because: VM_BUG_ON_PAGE(page_ref_count(page) == 0) ------------[ cut here ]------------ kernel BUG at /home/kas/linux/la57/include/linux/mm.h:499! invalid opcode: 0000 [#1] PREEMPT SMP CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.18.0-rc2-00037-g6f0d349d922b-dirty #58 Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015 RIP: 0010:__free_pages+0x28/0x30 Code: 00 00 8b 47 34 85 c0 74 15 f0 ff 4f 34 75 09 85 f6 74 06 e9 ca d8 ff ff c3 e9 64 ff ff ff 48 RSP: 0000:ffffffff9a403e90 EFLAGS: 00000246 RAX: 000000000000003e RBX: ffffffff9a41d000 RCX: 0000000000000002 RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00000000ffffffff RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000000 R12: ffff9092af089000 R13: ffffffff9a598a80 R14: 0000000000000001 R15: 0000000000000001 FS: 0000000000000000(0000) GS:ffff9092bfc00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ffff9092bffff000 CR3: 0000000198e1d000 CR4: 00000000000006b0 Call Trace: efi_call_phys_epilog+0x17d/0x1bb efi_enter_virtual_mode+0x457/0x4ca start_kernel+0x443/0x4dc secondary_startup_64+0xb7/0xc0 Modules linked in: ---[ end trace 61e271260b11acdd ]---I'll send patch for
efi_call_phys_epilog().Quoted 3 lines
Fixes 98219dda2ab5 ("x86/mm: Fold p4d page table layer at runtime")Signed-off-by Andrey Ryabinin <[email protected]>Reviewed-by Kirill A. Shutemov <[email protected]>Cc [email protected] # 4.17Quoted 18 lines
--- arch/x86/include/asm/pgalloc.h | 3 +++ 1 file changed, 3 insertions(+)arch/x86/include/asm/pgalloc.h +3
-0@@ -184,6 +184,9 @@ static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr)static inline void p4d_free (struct mm_struct *mm,p4d_t *p4d) { +if (!pgtable_l5_enabled ()) +return ; +BUG_ON ((unsigned long )p4d & (PAGE_SIZE -1 ));free_page ((unsigned long )p4d); }arch/x86/platform/efi/efi_64.c +0
-0@@ -166,14 +166,14 @@ void __init efi_call_phys_epilog(pgd_t *save_pgd) pgd =pgd_offset_k (pgd_idx *PGDIR_SIZE );set_pgd (pgd_offset_k (pgd_idx *PGDIR_SIZE ), save_pgd[pgd_idx]);Quoted 6 lines
arch/x86/platform/efi/efi_64.c void __init efi_call_phys_epilog(pgd_t *save_pgd) · line 170
Quoted 3 lines
arch/x86/platform/efi/efi_64.c +1
-1-if (!(pgd_val (*pgd) & _PAGE_PRESENT)) +if (!pgd_present (*pgd))continue ;I may not understand boot-time p4d folding. Here p4d is folded, why
pgd_present()need always return 1 if p4d folded?
Quoted 6 lines
Yeah. This is confusing. Basically, we pretend that p4d level consist of 1 entry, with the same value as pgd above. We say that pgd is always present and all checks happens on p4d level.
It's not specific to boot-time folding. Compilet-time folding doing the same. See
include/asm-generic/pgtable-nop4d.h(and the rest -nop?d.h)
Thanks for telling, Kirill. Then the next p4d_present will do the real _PAGE_PRESENT checking. If so, I think this is a good fix.
tip-bot for Andrey Ryabinin
merge bot[tip:x86/urgent] x86/mm: Don't free P4D table when it is folded at runtime
In reply to Andrey Ryabininx86/mm: Don't free P4D table when it is folded at runtime
Commit-ID: 0e311d237d7f3022b7dafb639b42541bfb42fe94 Gitweb: https://git.kernel.org/tip/0e311d237d7f3022b7dafb639b42541bfb42fe94 Author: Andrey Ryabinin <[email protected]> AuthorDate: Mon, 25 Jun 2018 13:24:27 +0300 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 26 Jun 2018 09:21:48 +0200
x86/mm: Don't free P4D table when it is folded at runtime
When the P4D page table layer is folded at runtime, the p4d_free() should do nothing, the same as in <asm-generic/pgtable-nop4d.h>.
It seems this bug should cause double-free in efi_call_phys_epilog(), but I don't know how to trigger that code path, so I can't confirm that by testing.
--- arch/x86/include/asm/pgalloc.h | 3 +++ 1 file changed, 3 insertions(+)
arch/x86/include/asm/pgalloc.h +3 -0
@@ -184,6 +184,9 @@ static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr)static inline void p4d_free (struct mm_struct *mm,p4d_t *p4d) { +if (!pgtable_l5_enabled ()) +return ; +BUG_ON ((unsigned long )p4d & (PAGE_SIZE -1 ));free_page ((unsigned long )p4d); }