x86/efi: unconditionally hold the whole low-1MB memory regions
Subsystems (5)
PATCH v2 1/1 x86/efi: unconditionally hold the whole low-1MB memory regions
Some sub-1MB memory regions may be reserved by EFI boot services, and the memory regions will be released later in the efi_free_boot_services().
Currently, always reserve all sub-1MB memory regions when the crashkernel option is specified, but unfortunately EFI boot services may have already reserved some sub-1MB memory regions before the crash_reserve_low_1M() is called, which makes that the crash_reserve_low_1M() only own the remaining sub-1MB memory regions, not all sub-1MB memory regions, because, subsequently EFI boot services will free its own sub-1MB memory regions. Eventually, DMA will be able to allocate memory from the sub-1MB area and cause the following error:
crash> kmem -s |grep invalid
kmem: dma-kmalloc-512: slab: ffffd52c40001900 invalid freepointer: ffff9403c0067300
kmem: dma-kmalloc-512: slab: ffffd52c40001900 invalid freepointer: ffff9403c0067300 crash> vtop ffff9403c0067300
VIRTUAL PHYSICALffff9403c0067300 67300 --->The physical address falls into this range [0x0000000000063000-0x000000000008efff]
kernel debugging log:
...
[ 0.008927] memblock_reserve: [0x0000000000010000-0x0000000000013fff] efi_reserve_boot_services+0x85/0xd0 [ 0.008930] memblock_reserve: [0x0000000000063000-0x000000000008efff] efi_reserve_boot_services+0x85/0xd0 ... [ 0.009425] memblock_reserve: [0x0000000000000000-0x00000000000fffff] crash_reserve_low_1M+0x2c/0x49 ... [ 0.010586] Zone ranges: [ 0.010587] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.010589] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.010591] Normal [mem 0x0000000100000000-0x0000000c7fffffff] [ 0.010593] Device empty ... [ 8.814894] __memblock_free_late: [0x0000000000063000-0x000000000008efff] efi_free_boot_services+0x14b/0x23b [ 8.815793] __memblock_free_late: [0x0000000000010000-0x0000000000013fff] efi_free_boot_services+0x14b/0x23b
To fix the above issues, let's hold the whole low-1M memory regions unconditionally in the efi_free_boot_services().
---
Background(copy from bhe's comment in the patch v1):
Kdump kernel also need go through real mode code path during bootup. It is not different than normal kernel except that it skips the firmware resetting. So kdump kernel needs low 1M as system RAM just as normal kernel does. Here we reserve the whole low 1M with memblock_reserve() to avoid any later kernel or driver data reside in this area. Otherwise, we need dump the content of this area to vmcore. As we know, when crash happened, the old memory of 1st kernel should be untouched until vmcore dumping read out its content. Meanwhile, kdump kernel need reuse low 1M. In the past, we used a back up region to copy out the low 1M area, and map the back up region into the low 1M area in vmcore elf file. In 6f599d84231fd27 ("x86/kdump: Always reserve the low 1M when the crashkernel option is specified"), we changed to lock the whole low 1M to avoid writting any kernel data into, like this we can skip this area when dumping vmcore.
Above is why we try to memblock reserve the whole low 1M. We don't want to use it, just don't want anyone to use it in 1st kernel.
arch/x86/platform/efi/quirks.c | 32 +++++++++++++++----------------- 1 file changed, 15 insertions(+), 17 deletions(-)
arch/x86/platform/efi/quirks.c +15 -17
@@ -11,6 +11,7 @@#include <linux/memblock.h> #include <linux/acpi.h> #include <linux/dmi.h> +#include <linux/sizes.h> #include <asm/e820/api.h> #include <asm/efi.h>
@@ -409,7 +410,7 @@ void __init efi_free_boot_services(void)for_each_efi_memory_desc (md) {unsigned long long start = md->phys_addr;unsigned long long size = md->num_pages <<EFI_PAGE_SHIFT ; -size_t rm_size; +unsigned long long end = start + size;if (md->type !=EFI_BOOT_SERVICES_CODE && md->type !=EFI_BOOT_SERVICES_DATA ) {
@@ -431,23 +432,20 @@ void __init efi_free_boot_services(void)efi_unmap_pages (md);/* -* Nasty quirk: if all sub-1MB memory is used for boot -* services, we can get here without having allocated the -* real mode trampoline. It's too late to hand boot services -* memory back to the memblock allocator, so instead -* try to manually allocate the trampoline if needed. -* -* I've seen this on a Dell XPS 13 9350 with firmware -* 1.4.4 with SGX enabled booting Linux via Fedora 24's -* grub2-efi on a hard disk. (And no, I don't know why -* this happened, but Linux should still try to boot rather -* panicking early.) +* The sub-1MB memory may be within the range[0, SZ_1M] +* or across the low-1M memory boundary. Let's handle +* these two cases and hold the whole low-1M memory +* unconditionally. */ - rm_size =real_mode_size_needed (); -if (rm_size && (start + rm_size) < (1 <<20 ) && size >= rm_size) { -set_real_mode_mem (start); - start += rm_size; - size -= rm_size; +if (start <SZ_1M ) { +/* Within the range[0, SZ_1M] */ +if (end <=SZ_1M ) +continue ; +else { +/* Across the low-1M memory boundary */ + size -= (SZ_1M - start); + start =SZ_1M ; + } }memblock_free_late (start, size);
Signature · 7 lines
2.17.1 _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Borislav Petkov
maintainer+ Mike.
Quoted 113 lines · Lianbo Jiang
Some sub-1MB memory regions may be reserved by EFI boot services, and the memory regions will be released later in the
efi_free_boot_services().Currently, always reserve all sub-1MB memory regions when the crashkernel option is specified, but unfortunately EFI boot services may have already reserved some sub-1MB memory regions before the
crash_reserve_low_1M()is called, which makes that thecrash_reserve_low_1M()only own the remaining sub-1MB memory regions, not all sub-1MB memory regions, because, subsequently EFI boot services will free its own sub-1MB memory regions. Eventually, DMA will be able to allocate memory from the sub-1MB area and cause the following error:crash> kmem -s |grep invalid
kmem: dma-kmalloc-512: slab:ffffd52c40001900invalid freepointer:ffff9403c0067300
kmem: dma-kmalloc-512: slab:ffffd52c40001900invalid freepointer:ffff9403c0067300crash> vtopffff9403c0067300
VIRTUAL PHYSICALffff9403c006730067300 --->The physical address falls into this range [0x0000000000063000-0x000000000008efff]kernel debugging log:
...[ 0.008927] memblock_reserve: [0x0000000000010000-0x0000000000013fff] efi_reserve_boot_services+0x85/0xd0 [ 0.008930] memblock_reserve: [0x0000000000063000-0x000000000008efff] efi_reserve_boot_services+0x85/0xd0 ... [ 0.009425] memblock_reserve: [0x0000000000000000-0x00000000000fffff] crash_reserve_low_1M+0x2c/0x49 ... [ 0.010586] Zone ranges: [ 0.010587] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.010589] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.010591] Normal [mem 0x0000000100000000-0x0000000c7fffffff] [ 0.010593] Device empty ... [ 8.814894] __memblock_free_late: [0x0000000000063000-0x000000000008efff] efi_free_boot_services+0x14b/0x23b [ 8.815793] __memblock_free_late: [0x0000000000010000-0x0000000000013fff] efi_free_boot_services+0x14b/0x23bTo fix the above issues, let's hold the whole low-1M memory regions unconditionally in the
efi_free_boot_services().Signed-off-by Lianbo Jiang <[email protected]>---
Background(copy from bhe's comment in the patch v1):Kdump kernel also need go through real mode code path during bootup. It is not different than normal kernel except that it skips the firmware resetting. So kdump kernel needs low 1M as system RAM just as normal kernel does. Here we reserve the whole low 1M with
memblock_reserve()to avoid any later kernel or driver data reside in this area. Otherwise, we need dump the content of this area to vmcore. As we know, when crash happened, the old memory of 1st kernel should be untouched until vmcore dumping read out its content. Meanwhile, kdump kernel need reuse low 1M. In the past, we used a back up region to copy out the low 1M area, and map the back up region into the low 1M area in vmcore elf file. In6f599d84231fd27("x86/kdump: Always reserve the low 1M when the crashkernel option is specified"), we changed to lock the whole low 1M to avoid writting any kernel data into, like this we can skip this area when dumping vmcore.Above is why we try to memblock reserve the whole low 1M. We don't want to use it, just don't want anyone to use it in 1st kernel.
arch/x86/platform/efi/quirks.c | 32 +++++++++++++++----------------- 1 file changed, 15 insertions(+), 17 deletions(-)arch/x86/platform/efi/quirks.c +8
-17@@ -11,6 +11,7 @@#include <linux/memblock.h> #include <linux/acpi.h> #include <linux/dmi.h> +#include <linux/sizes.h> #include <asm/e820/api.h> #include <asm/efi.h> @@ -409,7 +410,7 @@ void __init efi_free_boot_services(void)for_each_efi_memory_desc (md) {unsigned long long start = md->phys_addr;unsigned long long size = md->num_pages <<EFI_PAGE_SHIFT ; -size_t rm_size; +unsigned long long end = start + size;if (md->type !=EFI_BOOT_SERVICES_CODE && md->type !=EFI_BOOT_SERVICES_DATA ) {@@ -431,23 +432,20 @@ void __init efi_free_boot_services(void)efi_unmap_pages (md);/* -* Nasty quirk: if all sub-1MB memory is used for boot -* services, we can get here without having allocated the -* real mode trampoline. It's too late to hand boot services -* memory back to the memblock allocator, so instead -* try to manually allocate the trampoline if needed. -* -* I've seen this on a Dell XPS 13 9350 with firmware -* 1.4.4 with SGX enabled booting Linux via Fedora 24's -* grub2-efi on a hard disk. (And no, I don't know why -* this happened, but Linux should still try to boot rather -* panicking early.) +* The sub-1MB memory may be within the range[0, SZ_1M] +* or across the low-1M memory boundary. Let's handle +* these two cases and hold the whole low-1M memory +* unconditionally. */ - rm_size =real_mode_size_needed (); -if (rm_size && (start + rm_size) < (1 <<20 ) && size >= rm_size) { -set_real_mode_mem (start); - start += rm_size; - size -= rm_size; +if (start <SZ_1M ) { +/* Within the range[0, SZ_1M] */
arch/x86/platform/efi/quirks.c void __init efi_free_boot_services(void) · line 451
Quoted 10 lines
arch/x86/platform/efi/quirks.c +7
-0+if (end <=SZ_1M ) +continue ; +else { +/* Across the low-1M memory boundary */ + size -= (SZ_1M - start); + start =SZ_1M ; + } }memblock_free_late (start, size);
I don't think this will be needed when this here happens:
https://lkml.kernel.org/r/YK%[email protected]
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteMike Rapoport
maintainerQuoted 103 lines · Borislav Petkov
+ Mike.
Quoted 100 lines · Lianbo Jiang
Some sub-1MB memory regions may be reserved by EFI boot services, and the memory regions will be released later in the
efi_free_boot_services().Currently, always reserve all sub-1MB memory regions when the crashkernel option is specified, but unfortunately EFI boot services may have already reserved some sub-1MB memory regions before the
crash_reserve_low_1M()is called, which makes that thecrash_reserve_low_1M()only own the remaining sub-1MB memory regions, not all sub-1MB memory regions, because, subsequently EFI boot services will free its own sub-1MB memory regions. Eventually, DMA will be able to allocate memory from the sub-1MB area and cause the following error:crash> kmem -s |grep invalid
kmem: dma-kmalloc-512: slab:ffffd52c40001900invalid freepointer:ffff9403c0067300
kmem: dma-kmalloc-512: slab:ffffd52c40001900invalid freepointer:ffff9403c0067300crash> vtopffff9403c0067300
VIRTUAL PHYSICALffff9403c006730067300 --->The physical address falls into this range [0x0000000000063000-0x000000000008efff]kernel debugging log:
...[ 0.008927] memblock_reserve: [0x0000000000010000-0x0000000000013fff] efi_reserve_boot_services+0x85/0xd0 [ 0.008930] memblock_reserve: [0x0000000000063000-0x000000000008efff] efi_reserve_boot_services+0x85/0xd0 ... [ 0.009425] memblock_reserve: [0x0000000000000000-0x00000000000fffff] crash_reserve_low_1M+0x2c/0x49 ... [ 0.010586] Zone ranges: [ 0.010587] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.010589] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.010591] Normal [mem 0x0000000100000000-0x0000000c7fffffff] [ 0.010593] Device empty ... [ 8.814894] __memblock_free_late: [0x0000000000063000-0x000000000008efff] efi_free_boot_services+0x14b/0x23b [ 8.815793] __memblock_free_late: [0x0000000000010000-0x0000000000013fff] efi_free_boot_services+0x14b/0x23bTo fix the above issues, let's hold the whole low-1M memory regions unconditionally in the
efi_free_boot_services().Signed-off-by Lianbo Jiang <[email protected]>---
Background(copy from bhe's comment in the patch v1):Kdump kernel also need go through real mode code path during bootup. It is not different than normal kernel except that it skips the firmware resetting. So kdump kernel needs low 1M as system RAM just as normal kernel does. Here we reserve the whole low 1M with
memblock_reserve()to avoid any later kernel or driver data reside in this area. Otherwise, we need dump the content of this area to vmcore. As we know, when crash happened, the old memory of 1st kernel should be untouched until vmcore dumping read out its content. Meanwhile, kdump kernel need reuse low 1M. In the past, we used a back up region to copy out the low 1M area, and map the back up region into the low 1M area in vmcore elf file. In6f599d84231fd27("x86/kdump: Always reserve the low 1M when the crashkernel option is specified"), we changed to lock the whole low 1M to avoid writting any kernel data into, like this we can skip this area when dumping vmcore.Above is why we try to memblock reserve the whole low 1M. We don't want to use it, just don't want anyone to use it in 1st kernel.
arch/x86/platform/efi/quirks.c | 32 +++++++++++++++----------------- 1 file changed, 15 insertions(+), 17 deletions(-)arch/x86/platform/efi/quirks.c +2
-12@@ -11,6 +11,7 @@#include <linux/memblock.h> #include <linux/acpi.h> #include <linux/dmi.h> +#include <linux/sizes.h> #include <asm/e820/api.h> #include <asm/efi.h> @@ -409,7 +410,7 @@ void __init efi_free_boot_services(void)for_each_efi_memory_desc (md) {unsigned long long start = md->phys_addr;unsigned long long size = md->num_pages <<EFI_PAGE_SHIFT ; -size_t rm_size; +unsigned long long end = start + size;if (md->type !=EFI_BOOT_SERVICES_CODE && md->type !=EFI_BOOT_SERVICES_DATA ) {@@ -431,23 +432,20 @@ void __init efi_free_boot_services(void)efi_unmap_pages (md);/* -* Nasty quirk: if all sub-1MB memory is used for boot -* services, we can get here without having allocated the -* real mode trampoline. It's too late to hand boot services -* memory back to the memblock allocator, so instead -* try to manually allocate the trampoline if needed. -* -* I've seen this on a Dell XPS 13 9350 with firmware -* 1.4.4 with SGX enabled booting Linux via Fedora 24's -* grub2-efi on a hard disk. (And no, I don't know why -* this happened, but Linux should still try to boot rather -* panicking early.)
Quoted 8 lines
arch/x86/platform/efi/quirks.c void __init efi_free_boot_services(void) · line 448
Quoted 8 lines
arch/x86/platform/efi/quirks.c +4
-3+* The sub-1MB memory may be within the range[0, SZ_1M] +* or across the low-1M memory boundary. Let's handle +* these two cases and hold the whole low-1M memory +* unconditionally. */ - rm_size =real_mode_size_needed (); -if (rm_size && (start + rm_size) < (1 <<20 ) && size >= rm_size) { -set_real_mode_mem (start);
This will restore the issue with Dell XPS described above (presuming it's still relevant with recent kernels).
Quoted 13 lines
Quoted 13 lines
arch/x86/platform/efi/quirks.c +9
-2- start += rm_size; - size -= rm_size; +if (start <SZ_1M ) { +/* Within the range[0, SZ_1M] */ +if (end <=SZ_1M ) +continue ; +else { +/* Across the low-1M memory boundary */ + size -= (SZ_1M - start); + start =SZ_1M ; + } }
Quoted 6 lines
arch/x86/platform/efi/quirks.c void __init efi_free_boot_services(void) · line 451
Quoted 2 lines
arch/x86/platform/efi/quirks.c +0
-0memblock_free_late (start, size);I don't think this will be needed when this here happens:
Right, but TBH, I didn't update efi_free_boot_services() in my initial version. I've added similar change there now and I'm waiting now to see if kbuild is happy with this:
https://git.kernel.org/pub/scm/linux/kernel/git/rppt/linux.git/log/?h=x86/reservelow
Signature · 7 lines
Sincerely yours, Mike. _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Borislav Petkov
maintainerQuoted 5 lines · Mike Rapoport
Right, but TBH, I didn't update
efi_free_boot_services()in my initial version. I've added similar change there now and I'm waiting now to see if kbuild is happy with this:https://git.kernel.org/pub/scm/linux/kernel/git/rppt/linux.git/log/?h=x86/reservelow
Right, also I'm guessing that first patch should be
as there was one report with failing boot, right?
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteA message that is not in the archive (sent off-list or not yet received): CANU+ZyeFxeGXhxDx_ojB_hmiD-b8aTkqJcmc4fBHAS39vS7ZiQ@mail.gmail.com
Mike Rapoport
maintainerQuoted 21 lines · lijiang
Thank you for the information, Boris and Mike.
BTW: I just noticed that Mike's patch is incorrect, maybe it's a typo:
arch/x86/platform/efi/quirks.c +9
-0@@ -450,6 +450,18 @@ void __init efi_free_boot_services(void) size -= rm_size; } +/* +* Don't free memory under 1M for two reasons: +* - BIOS might clobber it +* - Crash kernel needs it to be reserved +*/ +if (start + size <SZ_1M ) +continue ; +if (start <SZ_1M ) { + size -= (start -SZ_1M );^^^^^^^^^^^^^^^^^^^^^^^^
Quoted 1 line
It looks like: size -= (
SZ_1M- start);
Right, thanks!
Quoted 9 lines
+ start =
SZ_1M;
+ }
+memblock_free_late(start, size);
}Mike's patch link:
https://git.kernel.org/pub/scm/linux/kernel/git/rppt/linux.git/commit/?h=x86/
reservelow&id=479fb34676ac448529b605854cf48c007e796ccd
Signature · 7 lines
Sincerely yours, Mike. _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Mike Rapoport
maintainerQuoted 5 lines · Borislav Petkov
Quoted 4 lines · Mike Rapoport
Right, but TBH, I didn't update
efi_free_boot_services()in my initial version. I've added similar change there now and I'm waiting now to see if kbuild is happy with this:
Quoted 7 lines
Right, also I'm guessing that first patch should be
Cc <[email protected]>as there was one report with failing boot, right?
Hmm, why?
The regression is from v5.13-rc1, isn't it?
Signature · 7 lines
Sincerely yours, Mike. _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Borislav Petkov
maintainerQuoted 2 lines · Mike Rapoport
Hmm, why?
The regression is from v5.13-rc1, isn't it?
Ah ok
a799c2bd29d1 ("x86/setup: Consolidate early memory reservations")
went into -rc1.
Thx.
Signature · 9 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec