Skip to contentSkip to conversation

Question about Address Range Validation in Crash Kernel Allocation

DISCUSSION chenhaixiang (A) 4 participants 8 messages Mar 21, 2024 – Mar 22, 2024

Subsystems (4)
lore
Share & tools
lore
#0 lore raw
Quoted 38 lines
Quoted 2 lines

I'm sorry for the delay. Here are some details from the boot log and

/proc/iomem:

Quoted 2 lines

The Boot log:

[    0.000000] Linux version 6.8.0 ([email protected]) (gcc (GCC)

10.3.1, GNU ld (GNU Binutils) 2.37) #3 SMP PREEMPT_DYNAMIC Wed Mar 20 11:46:11 UTC 2024

Quoted 1 line
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-6.8.0

root=/dev/mapper/root ro crashkernel=512M resume=/dev/mapper/swap rd.lvm.lv=root rd.lvm.lv=swap crash_kexec_post_notifiers softlockup_panic=1 reserve_kbox_mem=16M fsck.mode=auto fsck.repair=yes panic=3 nmi_watchdog=1 quiet rd.shell=0 memblock=debug efi=debug console=ttyS0,115200n8 console=tty0
......snip...

Quoted 1 line
[    0.022622] memblock_phys_alloc_range: 536870912 bytes align=0x1000000

from=0x0000000000000000 max_addr=0x0000000100000000 reserve_crashkernel_generic+0x7c/0x220

Quoted 1 line
[    0.022628] memblock_phys_alloc_range: 536870912 bytes align=0x1000000

from=0x0000000100000000 max_addr=0x0000400000000000 reserve_crashkernel_generic+0x7c/0x220

Quoted 1 line
[    0.022632] memblock_reserve: [0x000000c01f000000-0x000000c03effffff]

memblock_alloc_range_nid+0xee/0x170

Quoted 1 line
[    0.022634] memblock_phys_alloc_range: 268435456 bytes align=0x1000000

from=0x0000000000000000 max_addr=0x0000000100000000 reserve_crashkernel_generic+0x11d/0x220

Quoted 1 line
[    0.022638] memblock_reserve: [0x0000000049000000-0x0000000058ffffff]

memblock_alloc_range_nid+0xee/0x170

Quoted 1 line
[    0.022640] crashkernel low memory reserved: 0x49000000 - 0x59000000

(256 MB)

Quoted 1 line
[    0.022641] crashkernel reserved: 0x000000c01f000000 -

0x000000c03f000000 (512 MB)

Here, crashkernel,low is reserved in region: [0x49000000 - 0x59000000] (256 MB)

      crashkernel,high is reserved in region: [0x000000c01f000000 -

0x000000c03f000000] (512 MB) ......

Quoted 1 line
[    0.029839] memblock_reserve: [0x000000c03ffff740-0x000000c03fffff7f]

memblock_alloc_range_nid+0xee/0x170

Quoted 8 lines
Quoted 1 line
[    0.029843] e820: update [mem 0x53cbd000-0x53ccffff] usable ==>

reserved

Quoted 1 line
[    0.029861] TSC deadline timer available

Then here, region [0x53cbd000-0x53ccffff] is reserved in e820, and print abvoe "usable ==> reserved". This should be the step which prevents earlier reserved crashkernel,low from being added to iomem tree. I am not sure what triggered the e820 update.

Current analysis suggests that efi_reserve_boot_services() is causing the update of the e820 table.

Quoted 3 lines

How do you boot into your new 6.8.0 kernel? Used kexec -l to jump into the 2nd kernel, or reboot from bios/firmware boot up into 6.8.0?

It's reboot from bios boot up into 6.8.0. I attempted to revert the below patch,
and this time the conflicting segment "53cbd000-53ccffff" also appeared in the /proc/iomem of the 6.8 kernel.

2d4fd058-60efefff : System RAM
2d4fd058-58ffffff : System RAM

    49000000-58ffffff : Crash kernel
      53cbd000-53ccffff : Reserved

60eff000-704fefff : Reserved
--
93dd424000-93dd9fffff : Kernel bss
c01f000000-c03effffff : Crash kernel
d0000000000-d0fffffffff : PCI Bus 0000:00
d0000000000-d00001fffff : PCI Bus 0000:01

Quoted 8 lines

Reverting below commit should fix your problem, can you try it?

commit 4a693ce65b186fddc1a73621bd6f941e6e3eca21

Author: Huacai Chen <[email protected]>
Date:   Fri Dec 29 16:02:13 2023 +0800

kdump: defer the insertion of crashkernel resources

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#1 lore raw

Hi Baoquan,

Quoted 38 lines · chenhaixiang (A)
Quoted 38 lines
Quoted 2 lines

I'm sorry for the delay. Here are some details from the boot log and

/proc/iomem:

Quoted 2 lines

The Boot log:

[    0.000000] Linux version 6.8.0 ([email protected]) (gcc (GCC)

10.3.1, GNU ld (GNU Binutils) 2.37) #3 SMP PREEMPT_DYNAMIC Wed Mar 20 11:46:11 UTC 2024

Quoted 1 line
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-6.8.0

root=/dev/mapper/root ro crashkernel=512M resume=/dev/mapper/swap rd.lvm.lv=root rd.lvm.lv=swap crash_kexec_post_notifiers softlockup_panic=1 reserve_kbox_mem=16M fsck.mode=auto fsck.repair=yes panic=3 nmi_watchdog=1 quiet rd.shell=0 memblock=debug efi=debug console=ttyS0,115200n8 console=tty0
......snip...

Quoted 1 line
[    0.022622] memblock_phys_alloc_range: 536870912 bytes align=0x1000000

from=0x0000000000000000 max_addr=0x0000000100000000 reserve_crashkernel_generic+0x7c/0x220

Quoted 1 line
[    0.022628] memblock_phys_alloc_range: 536870912 bytes align=0x1000000

from=0x0000000100000000 max_addr=0x0000400000000000 reserve_crashkernel_generic+0x7c/0x220

Quoted 1 line
[    0.022632] memblock_reserve: [0x000000c01f000000-0x000000c03effffff]

memblock_alloc_range_nid+0xee/0x170

Quoted 1 line
[    0.022634] memblock_phys_alloc_range: 268435456 bytes align=0x1000000

from=0x0000000000000000 max_addr=0x0000000100000000 reserve_crashkernel_generic+0x11d/0x220

Quoted 1 line
[    0.022638] memblock_reserve: [0x0000000049000000-0x0000000058ffffff]

memblock_alloc_range_nid+0xee/0x170

Quoted 1 line
[    0.022640] crashkernel low memory reserved: 0x49000000 - 0x59000000

(256 MB)

Quoted 1 line
[    0.022641] crashkernel reserved: 0x000000c01f000000 -

0x000000c03f000000 (512 MB)

Here, crashkernel,low is reserved in region: [0x49000000 - 0x59000000] (256 MB)

      crashkernel,high is reserved in region: [0x000000c01f000000 -

0x000000c03f000000] (512 MB) ......

Quoted 1 line
[    0.029839] memblock_reserve: [0x000000c03ffff740-0x000000c03fffff7f]

memblock_alloc_range_nid+0xee/0x170

Quoted 8 lines
Quoted 8 lines
Quoted 1 line
[    0.029843] e820: update [mem 0x53cbd000-0x53ccffff] usable ==>

reserved

Quoted 1 line
[    0.029861] TSC deadline timer available

Then here, region [0x53cbd000-0x53ccffff] is reserved in e820, and print abvoe "usable ==> reserved". This should be the step which prevents earlier reserved crashkernel,low from being added to iomem tree. I am not sure what triggered the e820 update.

We added dump_stack () printing in efi_mem_reserve () and found that [0x53cbd000-0x53ccffff] was reserved by BGRT:

  [    0.032259] e820: update [mem 0x53cbd000-0x53ccffff] usable ==>
reserved
  [    0.032262] CPU: 0 PID: 0 Comm: swapper Not tainted
5.10.0-60.18.0.50.h820.eulerosv2r11.x86_64 #7
  [    0.032263] Hardware name: Huawei 2288H V5/BC11SPSCB0, BIOS 8.25
08/30/2022
  [    0.032264] Call Trace:
  [    0.032265]  ? dump_stack+0x57/0x6e
  [    0.032267]  ? bgrt_init+0xc2/0xc2
  [    0.032268]  ? __e820__range_update+0x7a/0x1d6
  [    0.032270]  ? bgrt_init+0xc2/0xc2
  [    0.032272]  ? bgrt_init+0xc2/0xc2
  [    0.032274]  ? efi_arch_mem_reserve+0x1a3/0x1d0
  [    0.032276]  ? efi_mem_reserve+0x2d/0x42
  [    0.032278]  ? acpi_parse_bgrt+0xa/0x11
  [    0.032279]  ? acpi_table_parse+0x86/0xbc
  [    0.032281]  ? acpi_boot_init+0x79/0xad
  [    0.032282]  ? setup_arch+0x835/0x954
  [    0.032284]  ? start_kernel+0x5d/0x455
  [    0.032286]  ? secondary_startup_64_no_verify+0xc2/0xcb

efi_reserve_boot_services() has reserved memory of type EFI_BOOT_SERVICES_CODE & EFI_BOOT_SERVICES_DATA before crashkernel. efi_bgrt_init() assumes that EFI_BOOT_SERVICES_DATA is not reserved by other modules. Then, the e820_table is directly updated, and the BGRT memory is reserved.

However, memblock_is_region_reserved() in efi_reserve_boot_services() returns true when the ranges only overlap.

already_reserved = memblock_is_region_reserved(start, size);

/*
* Because the following memblock_reserve() is paired
* with memblock_free_late() for this region in
* efi_free_boot_services(), we must be extremely
* careful not to reserve, and subsequently free,
* critical regions of memory (like the kernel image) or
* those regions that somebody else has already
* reserved.
*
* A good example of a critical region that must not be
* freed is page zero (first 4Kb of memory), which may
* contain boot services code/data but is marked
* E820_TYPE_RESERVED by trim_bios_range().
*/
if (!already_reserved) {
memblock_reserve(start, size);

/*
* If we are the first to reserve the region, no
* one else cares about it. We own it and can
* free it later.
*/
if (can_free_region(start, size))
continue;
}

As a result, some memory of EFI_BOOT_SERVICES_DATA is not reserved in advance. The subsequent crashkernel happens to reserve this portion of memory, which conflicts with BGRT.

Quoted 22 lines

Current analysis suggests that efi_reserve_boot_services() is causing the update of the e820 table.

Quoted 3 lines

How do you boot into your new 6.8.0 kernel? Used kexec -l to jump into the 2nd kernel, or reboot from bios/firmware boot up into 6.8.0?

It's reboot from bios boot up into 6.8.0. I attempted to revert the below patch,
and this time the conflicting segment "53cbd000-53ccffff" also appeared in the /proc/iomem of the 6.8 kernel.

2d4fd058-60efefff : System RAM
2d4fd058-58ffffff : System RAM

    49000000-58ffffff : Crash kernel
      53cbd000-53ccffff : Reserved

60eff000-704fefff : Reserved
--
93dd424000-93dd9fffff : Kernel bss
c01f000000-c03effffff : Crash kernel
d0000000000-d0fffffffff : PCI Bus 0000:00
d0000000000-d00001fffff : PCI Bus 0000:01

Quoted 3 lines

Reverting below commit should fix your problem, can you try it?

Quoted 8 lines
Quoted 5 lines

commit 4a693ce65b186fddc1a73621bd6f941e6e3eca21

Author: Huacai Chen <[email protected]>
Date:   Fri Dec 29 16:02:13 2023 +0800

kdump: defer the insertion of crashkernel resources

.

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#2 lore raw

Hi,

Quoted 81 lines · Li Huafei

Hi Baoquan,

Quoted 38 lines · chenhaixiang (A)
Quoted 38 lines
Quoted 2 lines

I'm sorry for the delay. Here are some details from the boot log and

/proc/iomem:

Quoted 2 lines

The Boot log:

[    0.000000] Linux version 6.8.0 ([email protected]) (gcc (GCC)

10.3.1, GNU ld (GNU Binutils) 2.37) #3 SMP PREEMPT_DYNAMIC Wed Mar 20 11:46:11 UTC 2024

Quoted 1 line
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-6.8.0

root=/dev/mapper/root ro crashkernel=512M resume=/dev/mapper/swap rd.lvm.lv=root rd.lvm.lv=swap crash_kexec_post_notifiers softlockup_panic=1 reserve_kbox_mem=16M fsck.mode=auto fsck.repair=yes panic=3 nmi_watchdog=1 quiet rd.shell=0 memblock=debug efi=debug console=ttyS0,115200n8 console=tty0
......snip...

Quoted 1 line
[    0.022622] memblock_phys_alloc_range: 536870912 bytes align=0x1000000

from=0x0000000000000000 max_addr=0x0000000100000000 reserve_crashkernel_generic+0x7c/0x220

Quoted 1 line
[    0.022628] memblock_phys_alloc_range: 536870912 bytes align=0x1000000

from=0x0000000100000000 max_addr=0x0000400000000000 reserve_crashkernel_generic+0x7c/0x220

Quoted 1 line
[    0.022632] memblock_reserve: [0x000000c01f000000-0x000000c03effffff]

memblock_alloc_range_nid+0xee/0x170

Quoted 1 line
[    0.022634] memblock_phys_alloc_range: 268435456 bytes align=0x1000000

from=0x0000000000000000 max_addr=0x0000000100000000 reserve_crashkernel_generic+0x11d/0x220

Quoted 1 line
[    0.022638] memblock_reserve: [0x0000000049000000-0x0000000058ffffff]

memblock_alloc_range_nid+0xee/0x170

Quoted 1 line
[    0.022640] crashkernel low memory reserved: 0x49000000 - 0x59000000

(256 MB)

Quoted 1 line
[    0.022641] crashkernel reserved: 0x000000c01f000000 -

0x000000c03f000000 (512 MB)

Here, crashkernel,low is reserved in region: [0x49000000 - 0x59000000] (256 MB)

      crashkernel,high is reserved in region: [0x000000c01f000000 -

0x000000c03f000000] (512 MB) ......

Quoted 1 line
[    0.029839] memblock_reserve: [0x000000c03ffff740-0x000000c03fffff7f]

memblock_alloc_range_nid+0xee/0x170

Quoted 8 lines
Quoted 8 lines
Quoted 1 line
[    0.029843] e820: update [mem 0x53cbd000-0x53ccffff] usable ==>

reserved

Quoted 1 line
[    0.029861] TSC deadline timer available

Then here, region [0x53cbd000-0x53ccffff] is reserved in e820, and print abvoe "usable ==> reserved". This should be the step which prevents earlier reserved crashkernel,low from being added to iomem tree. I am not sure what triggered the e820 update.

We added dump_stack () printing in efi_mem_reserve () and found that [0x53cbd000-0x53ccffff] was reserved by BGRT:

  [    0.032259] e820: update [mem 0x53cbd000-0x53ccffff] usable ==>
reserved
  [    0.032262] CPU: 0 PID: 0 Comm: swapper Not tainted
5.10.0-60.18.0.50.h820.eulerosv2r11.x86_64 #7
  [    0.032263] Hardware name: Huawei 2288H V5/BC11SPSCB0, BIOS 8.25
08/30/2022
  [    0.032264] Call Trace:
  [    0.032265]  ? dump_stack+0x57/0x6e
  [    0.032267]  ? bgrt_init+0xc2/0xc2
  [    0.032268]  ? __e820__range_update+0x7a/0x1d6
  [    0.032270]  ? bgrt_init+0xc2/0xc2
  [    0.032272]  ? bgrt_init+0xc2/0xc2
  [    0.032274]  ? efi_arch_mem_reserve+0x1a3/0x1d0
  [    0.032276]  ? efi_mem_reserve+0x2d/0x42
  [    0.032278]  ? acpi_parse_bgrt+0xa/0x11
  [    0.032279]  ? acpi_table_parse+0x86/0xbc
  [    0.032281]  ? acpi_boot_init+0x79/0xad
  [    0.032282]  ? setup_arch+0x835/0x954
  [    0.032284]  ? start_kernel+0x5d/0x455
  [    0.032286]  ? secondary_startup_64_no_verify+0xc2/0xcb

efi_reserve_boot_services() has reserved memory of type EFI_BOOT_SERVICES_CODE & EFI_BOOT_SERVICES_DATA before crashkernel. efi_bgrt_init() assumes that EFI_BOOT_SERVICES_DATA is not reserved by other modules. Then, the e820_table is directly updated, and the BGRT memory is reserved.

Quoted 4 lines

However, memblock_is_region_reserved() in efi_reserve_boot_services() returns true when the ranges only overlap.

     already_reserved = memblock_is_region_reserved(start, size);

Do you mean efi_reserve_boot_services is supposed to reserve the bgrt memory but it does not reserve it due to the region overlapping with some other reserved region? If so can you debug and find what exact memblock reserved region overlaps with the bgrt?

BTW, the previous email threads are weird, and not threading correctly, hard to find information.

Quoted 59 lines

/*
* Because the following memblock_reserve() is paired
* with memblock_free_late() for this region in
* efi_free_boot_services(), we must be extremely
* careful not to reserve, and subsequently free,
* critical regions of memory (like the kernel image) or
* those regions that somebody else has already
* reserved.
*
* A good example of a critical region that must not be
* freed is page zero (first 4Kb of memory), which may
* contain boot services code/data but is marked
* E820_TYPE_RESERVED by trim_bios_range().
*/
if (!already_reserved) {
memblock_reserve(start, size);

/*
* If we are the first to reserve the region, no
* one else cares about it. We own it and can
* free it later.
*/
if (can_free_region(start, size))
continue;
}

As a result, some memory of EFI_BOOT_SERVICES_DATA is not reserved in advance. The subsequent crashkernel happens to reserve this portion of memory, which conflicts with BGRT.

Quoted 28 lines

Current analysis suggests that efi_reserve_boot_services() is causing the update of the e820 table.

Quoted 3 lines

How do you boot into your new 6.8.0 kernel? Used kexec -l to jump into the 2nd kernel, or reboot from bios/firmware boot up into 6.8.0?

It's reboot from bios boot up into 6.8.0. I attempted to revert the below patch,
and this time the conflicting segment "53cbd000-53ccffff" also appeared in the /proc/iomem of the 6.8 kernel.

2d4fd058-60efefff : System RAM
2d4fd058-58ffffff : System RAM

    49000000-58ffffff : Crash kernel
      53cbd000-53ccffff : Reserved

60eff000-704fefff : Reserved
--
93dd424000-93dd9fffff : Kernel bss
c01f000000-c03effffff : Crash kernel
d0000000000-d0fffffffff : PCI Bus 0000:00
d0000000000-d00001fffff : PCI Bus 0000:01

Quoted 8 lines

Reverting below commit should fix your problem, can you try it?

commit 4a693ce65b186fddc1a73621bd6f941e6e3eca21

Author: Huacai Chen <[email protected]>
Date:   Fri Dec 29 16:02:13 2023 +0800

kdump: defer the insertion of crashkernel resources

Quoted 7 lines
Quoted 2 lines

.

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#3 lore raw
Quoted 87 lines · Dave Young

Hi,

Quoted 84 lines · Li Huafei

Hi Baoquan,

Quoted 38 lines · chenhaixiang (A)
Quoted 38 lines
Quoted 2 lines

I'm sorry for the delay. Here are some details from the boot log and

/proc/iomem:

Quoted 2 lines

The Boot log:

[    0.000000] Linux version 6.8.0 ([email protected]) (gcc (GCC)

10.3.1, GNU ld (GNU Binutils) 2.37) #3 SMP PREEMPT_DYNAMIC Wed Mar 20 11:46:11 UTC 2024

Quoted 1 line
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-6.8.0

root=/dev/mapper/root ro crashkernel=512M resume=/dev/mapper/swap rd.lvm.lv=root rd.lvm.lv=swap crash_kexec_post_notifiers softlockup_panic=1 reserve_kbox_mem=16M fsck.mode=auto fsck.repair=yes panic=3 nmi_watchdog=1 quiet rd.shell=0 memblock=debug efi=debug console=ttyS0,115200n8 console=tty0
......snip...

Quoted 1 line
[    0.022622] memblock_phys_alloc_range: 536870912 bytes align=0x1000000

from=0x0000000000000000 max_addr=0x0000000100000000 reserve_crashkernel_generic+0x7c/0x220

Quoted 1 line
[    0.022628] memblock_phys_alloc_range: 536870912 bytes align=0x1000000

from=0x0000000100000000 max_addr=0x0000400000000000 reserve_crashkernel_generic+0x7c/0x220

Quoted 1 line
[    0.022632] memblock_reserve: [0x000000c01f000000-0x000000c03effffff]

memblock_alloc_range_nid+0xee/0x170

Quoted 1 line
[    0.022634] memblock_phys_alloc_range: 268435456 bytes align=0x1000000

from=0x0000000000000000 max_addr=0x0000000100000000 reserve_crashkernel_generic+0x11d/0x220

Quoted 1 line
[    0.022638] memblock_reserve: [0x0000000049000000-0x0000000058ffffff]

memblock_alloc_range_nid+0xee/0x170

Quoted 1 line
[    0.022640] crashkernel low memory reserved: 0x49000000 - 0x59000000

(256 MB)

Quoted 1 line
[    0.022641] crashkernel reserved: 0x000000c01f000000 -

0x000000c03f000000 (512 MB)

Here, crashkernel,low is reserved in region: [0x49000000 - 0x59000000] (256 MB)

      crashkernel,high is reserved in region: [0x000000c01f000000 -

0x000000c03f000000] (512 MB) ......

Quoted 1 line
[    0.029839] memblock_reserve: [0x000000c03ffff740-0x000000c03fffff7f]

memblock_alloc_range_nid+0xee/0x170

Quoted 8 lines
Quoted 8 lines
Quoted 1 line
[    0.029843] e820: update [mem 0x53cbd000-0x53ccffff] usable ==>

reserved

Quoted 1 line
[    0.029861] TSC deadline timer available

Then here, region [0x53cbd000-0x53ccffff] is reserved in e820, and print abvoe "usable ==> reserved". This should be the step which prevents earlier reserved crashkernel,low from being added to iomem tree. I am not sure what triggered the e820 update.

We added dump_stack () printing in efi_mem_reserve () and found that [0x53cbd000-0x53ccffff] was reserved by BGRT:

  [    0.032259] e820: update [mem 0x53cbd000-0x53ccffff] usable ==>
reserved
  [    0.032262] CPU: 0 PID: 0 Comm: swapper Not tainted
5.10.0-60.18.0.50.h820.eulerosv2r11.x86_64 #7
  [    0.032263] Hardware name: Huawei 2288H V5/BC11SPSCB0, BIOS 8.25
08/30/2022
  [    0.032264] Call Trace:
  [    0.032265]  ? dump_stack+0x57/0x6e
  [    0.032267]  ? bgrt_init+0xc2/0xc2
  [    0.032268]  ? __e820__range_update+0x7a/0x1d6
  [    0.032270]  ? bgrt_init+0xc2/0xc2
  [    0.032272]  ? bgrt_init+0xc2/0xc2
  [    0.032274]  ? efi_arch_mem_reserve+0x1a3/0x1d0
  [    0.032276]  ? efi_mem_reserve+0x2d/0x42
  [    0.032278]  ? acpi_parse_bgrt+0xa/0x11
  [    0.032279]  ? acpi_table_parse+0x86/0xbc
  [    0.032281]  ? acpi_boot_init+0x79/0xad
  [    0.032282]  ? setup_arch+0x835/0x954
  [    0.032284]  ? start_kernel+0x5d/0x455
  [    0.032286]  ? secondary_startup_64_no_verify+0xc2/0xcb

efi_reserve_boot_services() has reserved memory of type EFI_BOOT_SERVICES_CODE & EFI_BOOT_SERVICES_DATA before crashkernel. efi_bgrt_init() assumes that EFI_BOOT_SERVICES_DATA is not reserved by other modules. Then, the e820_table is directly updated, and the BGRT memory is reserved.

However, memblock_is_region_reserved() in efi_reserve_boot_services() returns true when the ranges only overlap.

Quoted 6 lines
Quoted 1 line
     already_reserved = memblock_is_region_reserved(start, size);

Do you mean efi_reserve_boot_services is supposed to reserve the bgrt memory but it does not reserve it due to the region overlapping with some other reserved region? If so can you debug and find what exact memblock reserved region overlaps with the bgrt?

Yes. I added the following debug print to efi_reserve_boot_services():

arch/x86/platform/efi/quirks.c +4 -0
@@ -339,6 +339,10 @@ void __init efi_reserve_boot_services(void)
 
                already_reserved = memblock_is_region_reserved(start, size);
 
+               pr_info("kdumpdebug: efi_reserve_boot_services start 0x%lu, "
+                       "size 0x%lx, type 0x%lx, already_reserved %d\n",
+                       start, size, md->type, already_reserved);
+
                /*
                 * Because the following memblock_reserve() is paired
                 * with memblock_free_late() for this region in

This memory [0x0000005976a018-0x00000005976abc7] is reserved here, which belongs to EFI_BOOT_SERVICES_DATA.

    [    0.000000] memblock_reserve: [0x000000005976a018-0x000000005976abc7] efi_memattr_init+0x51/0xa0
It falls in the following range
    [    0.000000] efi: mem22: [Boot Data   |   |  |  |  |  |  |  |  |  |   |WB|WT|WC|UC] range=[0x0000000051329000-0x000000005cefefff] (187MB)

in efi_reserve_boot_services(), [0x0000005132900-0x00000005cefeff] will not be fully reserved because [0x0000005976a018-0x0000005976abc7] has already been reserved and overlaps with [0x0000005976a018-0x0000005976abc7]

    [    0.021316] efi: kdumpdebug: efi_reserve_boot_services start 0x51329000, size 0xbbd6000, type 0x4, already_reserved 1

It is not reserved by memblock, this free memory region is allocated by crashkernel

    [    0.022597] crashkernel low memory reserved: 0x49000000 - 0x59000000 (256 MB)
    [    0.022599] crashkernel reserved: 0x000000c01f000000 - 0x000000c03f000000 (512 MB)

In efi_bgrt_init (), it is assumed that the memory of the EFI_BOOT_SERVICES_DATA type has been successfully reserved. Therefore, the address in the range is directly used. As a result, the memory overlaps with the crashkernel region.

    [    0.029694] e820: update [mem 0x53cbd000-0x53ccffff] usable ==> reserved
Quoted 3 lines

BTW, the previous email threads are weird, and not threading correctly, hard to find information.

It should be because the log content is too large and has been put on hold. In my previous email, I received a prompt:

The reason it is being held:

Message body is too big: 248998 bytes with a limit of 40 KB

Quoted 63 lines
Quoted 63 lines

/*
* Because the following memblock_reserve() is paired
* with memblock_free_late() for this region in
* efi_free_boot_services(), we must be extremely
* careful not to reserve, and subsequently free,
* critical regions of memory (like the kernel image) or
* those regions that somebody else has already
* reserved.
*
* A good example of a critical region that must not be
* freed is page zero (first 4Kb of memory), which may
* contain boot services code/data but is marked
* E820_TYPE_RESERVED by trim_bios_range().
*/
if (!already_reserved) {
memblock_reserve(start, size);

/*
* If we are the first to reserve the region, no
* one else cares about it. We own it and can
* free it later.
*/
if (can_free_region(start, size))
continue;
}

As a result, some memory of EFI_BOOT_SERVICES_DATA is not reserved in advance. The subsequent crashkernel happens to reserve this portion of memory, which conflicts with BGRT.

Quoted 30 lines

Current analysis suggests that efi_reserve_boot_services() is causing the update of the e820 table.

Quoted 3 lines

How do you boot into your new 6.8.0 kernel? Used kexec -l to jump into the 2nd kernel, or reboot from bios/firmware boot up into 6.8.0?

It's reboot from bios boot up into 6.8.0. I attempted to revert the below patch,
and this time the conflicting segment "53cbd000-53ccffff" also appeared in the /proc/iomem of the 6.8 kernel.

2d4fd058-60efefff : System RAM
2d4fd058-58ffffff : System RAM

    49000000-58ffffff : Crash kernel
      53cbd000-53ccffff : Reserved

60eff000-704fefff : Reserved
--
93dd424000-93dd9fffff : Kernel bss
c01f000000-c03effffff : Crash kernel
d0000000000-d0fffffffff : PCI Bus 0000:00
d0000000000-d00001fffff : PCI Bus 0000:01

Quoted 8 lines

Reverting below commit should fix your problem, can you try it?

commit 4a693ce65b186fddc1a73621bd6f941e6e3eca21

Author: Huacai Chen <[email protected]>
Date:   Fri Dec 29 16:02:13 2023 +0800

kdump: defer the insertion of crashkernel resources

.

Quoted 7 lines
Quoted 4 lines

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

.

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#4 lore raw
Quoted 103 lines · Li Huafei
Quoted 87 lines · Dave Young

Hi,

Quoted 84 lines · Li Huafei

Hi Baoquan,

Quoted 38 lines · chenhaixiang (A)
Quoted 38 lines
Quoted 2 lines

I'm sorry for the delay. Here are some details from the boot log and

/proc/iomem:

Quoted 2 lines

The Boot log:

[    0.000000] Linux version 6.8.0 ([email protected]) (gcc (GCC)

10.3.1, GNU ld (GNU Binutils) 2.37) #3 SMP PREEMPT_DYNAMIC Wed Mar 20 11:46:11 UTC 2024

Quoted 1 line
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-6.8.0

root=/dev/mapper/root ro crashkernel=512M resume=/dev/mapper/swap rd.lvm.lv=root rd.lvm.lv=swap crash_kexec_post_notifiers softlockup_panic=1 reserve_kbox_mem=16M fsck.mode=auto fsck.repair=yes panic=3 nmi_watchdog=1 quiet rd.shell=0 memblock=debug efi=debug console=ttyS0,115200n8 console=tty0
......snip...

Quoted 1 line
[    0.022622] memblock_phys_alloc_range: 536870912 bytes align=0x1000000

from=0x0000000000000000 max_addr=0x0000000100000000 reserve_crashkernel_generic+0x7c/0x220

Quoted 1 line
[    0.022628] memblock_phys_alloc_range: 536870912 bytes align=0x1000000

from=0x0000000100000000 max_addr=0x0000400000000000 reserve_crashkernel_generic+0x7c/0x220

Quoted 1 line
[    0.022632] memblock_reserve: [0x000000c01f000000-0x000000c03effffff]

memblock_alloc_range_nid+0xee/0x170

Quoted 1 line
[    0.022634] memblock_phys_alloc_range: 268435456 bytes align=0x1000000

from=0x0000000000000000 max_addr=0x0000000100000000 reserve_crashkernel_generic+0x11d/0x220

Quoted 1 line
[    0.022638] memblock_reserve: [0x0000000049000000-0x0000000058ffffff]

memblock_alloc_range_nid+0xee/0x170

Quoted 1 line
[    0.022640] crashkernel low memory reserved: 0x49000000 - 0x59000000

(256 MB)

Quoted 1 line
[    0.022641] crashkernel reserved: 0x000000c01f000000 -

0x000000c03f000000 (512 MB)

Here, crashkernel,low is reserved in region: [0x49000000 - 0x59000000] (256 MB)

      crashkernel,high is reserved in region: [0x000000c01f000000 -

0x000000c03f000000] (512 MB) ......

Quoted 1 line
[    0.029839] memblock_reserve: [0x000000c03ffff740-0x000000c03fffff7f]

memblock_alloc_range_nid+0xee/0x170

Quoted 8 lines
Quoted 8 lines
Quoted 1 line
[    0.029843] e820: update [mem 0x53cbd000-0x53ccffff] usable ==>

reserved

Quoted 1 line
[    0.029861] TSC deadline timer available

Then here, region [0x53cbd000-0x53ccffff] is reserved in e820, and print abvoe "usable ==> reserved". This should be the step which prevents earlier reserved crashkernel,low from being added to iomem tree. I am not sure what triggered the e820 update.

We added dump_stack () printing in efi_mem_reserve () and found that [0x53cbd000-0x53ccffff] was reserved by BGRT:

  [    0.032259] e820: update [mem 0x53cbd000-0x53ccffff] usable ==>
reserved
  [    0.032262] CPU: 0 PID: 0 Comm: swapper Not tainted
5.10.0-60.18.0.50.h820.eulerosv2r11.x86_64 #7
  [    0.032263] Hardware name: Huawei 2288H V5/BC11SPSCB0, BIOS 8.25
08/30/2022
  [    0.032264] Call Trace:
  [    0.032265]  ? dump_stack+0x57/0x6e
  [    0.032267]  ? bgrt_init+0xc2/0xc2
  [    0.032268]  ? __e820__range_update+0x7a/0x1d6
  [    0.032270]  ? bgrt_init+0xc2/0xc2
  [    0.032272]  ? bgrt_init+0xc2/0xc2
  [    0.032274]  ? efi_arch_mem_reserve+0x1a3/0x1d0
  [    0.032276]  ? efi_mem_reserve+0x2d/0x42
  [    0.032278]  ? acpi_parse_bgrt+0xa/0x11
  [    0.032279]  ? acpi_table_parse+0x86/0xbc
  [    0.032281]  ? acpi_boot_init+0x79/0xad
  [    0.032282]  ? setup_arch+0x835/0x954
  [    0.032284]  ? start_kernel+0x5d/0x455
  [    0.032286]  ? secondary_startup_64_no_verify+0xc2/0xcb

efi_reserve_boot_services() has reserved memory of type EFI_BOOT_SERVICES_CODE & EFI_BOOT_SERVICES_DATA before crashkernel. efi_bgrt_init() assumes that EFI_BOOT_SERVICES_DATA is not reserved by other modules. Then, the e820_table is directly updated, and the BGRT memory is reserved.

However, memblock_is_region_reserved() in efi_reserve_boot_services() returns true when the ranges only overlap.

Quoted 6 lines
Quoted 1 line
     already_reserved = memblock_is_region_reserved(start, size);

Do you mean efi_reserve_boot_services is supposed to reserve the bgrt memory but it does not reserve it due to the region overlapping with some other reserved region? If so can you debug and find what exact memblock reserved region overlaps with the bgrt?

Yes. I added the following debug print to efi_reserve_boot_services():

arch/x86/platform/efi/quirks.c void __init efi_reserve_boot_services(void) · line 348

Quoted 10 lines
arch/x86/platform/efi/quirks.c +4 -0
@@ -339,6 +339,10 @@ void __init efi_reserve_boot_services(void)
 
                already_reserved = memblock_is_region_reserved(start, size);
 
+               pr_info("kdumpdebug: efi_reserve_boot_services start 0x%lu, "
+                       "size 0x%lx, type 0x%lx, already_reserved %d\n",
+                       start, size, md->type, already_reserved);
+
                /*
                 * Because the following memblock_reserve() is paired
                 * with memblock_free_late() for this region in

It's great debugging and analysis, thanks you guys. Now there are several questions:

1) why memory region [0x5976a018-0x5976abc7] is reserved by memblock for efi_mem_attr_table. It's supposed to be outside of the EFI_BOOT_SERVICES_DATA area? We may need check here if it's a bug.

[    0.000000] random: crng init done
[    0.000000] memblock_reserve: [0x000000005976a018-0x000000005976abc7] efi_memattr_init+0x51/0xa0
Quoted 7 lines

This memory [0x0000005976a018-0x00000005976abc7] is reserved here, which belongs to EFI_BOOT_SERVICES_DATA.

    [    0.000000] memblock_reserve: [0x000000005976a018-0x000000005976abc7] efi_memattr_init+0x51/0xa0
It falls in the following range
    [    0.000000] efi: mem22: [Boot Data   |   |  |  |  |  |  |  |  |  |   |WB|WT|WC|UC] range=[0x0000000051329000-0x000000005cefefff] (187MB)

in efi_reserve_boot_services(), [0x0000005132900-0x00000005cefeff] will not be fully reserved because [0x0000005976a018-0x0000005976abc7] has already been reserved and overlaps with [0x0000005976a018-0x0000005976abc7]

2) Because efi_mem_attr_table memblock reserved [0x5976a018-0x5976abc7], the whole EFI_BOOT_SERVICES_DATA area [0x5132900-0x5cefeff] is not memblock reserved for later free. Excep of the small area, do we need still memblock reserve the remaining area, we may need check if this is a bug.

Quoted 11 lines
    [    0.021316] efi: kdumpdebug: efi_reserve_boot_services start 0x51329000, size 0xbbd6000, type 0x4, already_reserved 1

It is not reserved by memblock, this free memory region is allocated by crashkernel

    [    0.022597] crashkernel low memory reserved: 0x49000000 - 0x59000000 (256 MB)
    [    0.022599] crashkernel reserved: 0x000000c01f000000 - 0x000000c03f000000 (512 MB)

In efi_bgrt_init (), it is assumed that the memory of the EFI_BOOT_SERVICES_DATA type has been successfully reserved. Therefore, the address in the range is directly used. As a result, the memory overlaps with the crashkernel region.

(3) efi_bgrt_init() should be innocent because it's supposed to safely use the area according to the existing efi quirk handling.

(4) the deferring of adding crashh_low_res to iomem exposed the above efi issue. When we cancel the deferring of crashh_res inserting into iomem, we can see that the brgt area is reserved inside crashkernel region, that's problematic.

2d4fd058-60efefff : System RAM
2d4fd058-58ffffff : System RAM

    49000000-58ffffff : Crash kernel
      53cbd000-53ccffff : Reserved     <--- 

60eff000-704fefff : Reserved
--
93dd424000-93dd9fffff : Kernel bss
c01f000000-c03effffff : Crash kernel
d0000000000-d0fffffffff : PCI Bus 0000:00
d0000000000-d00001fffff : PCI Bus 0000:01

Quoted 84 lines
    [    0.029694] e820: update [mem 0x53cbd000-0x53ccffff] usable ==> reserved
Quoted 3 lines

BTW, the previous email threads are weird, and not threading correctly, hard to find information.

It should be because the log content is too large and has been put on hold. In my previous email, I received a prompt:

The reason it is being held:

Message body is too big: 248998 bytes with a limit of 40 KB

Quoted 70 lines
Quoted 60 lines

/*
* Because the following memblock_reserve() is paired
* with memblock_free_late() for this region in
* efi_free_boot_services(), we must be extremely
* careful not to reserve, and subsequently free,
* critical regions of memory (like the kernel image) or
* those regions that somebody else has already
* reserved.
*
* A good example of a critical region that must not be
* freed is page zero (first 4Kb of memory), which may
* contain boot services code/data but is marked
* E820_TYPE_RESERVED by trim_bios_range().
*/
if (!already_reserved) {
memblock_reserve(start, size);

/*
* If we are the first to reserve the region, no
* one else cares about it. We own it and can
* free it later.
*/
if (can_free_region(start, size))
continue;
}

As a result, some memory of EFI_BOOT_SERVICES_DATA is not reserved in advance. The subsequent crashkernel happens to reserve this portion of memory, which conflicts with BGRT.

Quoted 28 lines

Current analysis suggests that efi_reserve_boot_services() is causing the update of the e820 table.

Quoted 3 lines

How do you boot into your new 6.8.0 kernel? Used kexec -l to jump into the 2nd kernel, or reboot from bios/firmware boot up into 6.8.0?

It's reboot from bios boot up into 6.8.0. I attempted to revert the below patch,
and this time the conflicting segment "53cbd000-53ccffff" also appeared in the /proc/iomem of the 6.8 kernel.

2d4fd058-60efefff : System RAM
2d4fd058-58ffffff : System RAM

    49000000-58ffffff : Crash kernel
      53cbd000-53ccffff : Reserved

60eff000-704fefff : Reserved
--
93dd424000-93dd9fffff : Kernel bss
c01f000000-c03effffff : Crash kernel
d0000000000-d0fffffffff : PCI Bus 0000:00
d0000000000-d00001fffff : PCI Bus 0000:01

Quoted 8 lines

Reverting below commit should fix your problem, can you try it?

commit 4a693ce65b186fddc1a73621bd6f941e6e3eca21

Author: Huacai Chen <[email protected]>
Date:   Fri Dec 29 16:02:13 2023 +0800

kdump: defer the insertion of crashkernel resources

Quoted 7 lines
Quoted 2 lines

.

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

.

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#5 lore raw

Hi,

Quoted 119 lines · Baoquan He
Quoted 103 lines · Li Huafei
Quoted 87 lines · Dave Young

Hi,

Quoted 84 lines · Li Huafei

Hi Baoquan,

Quoted 38 lines · chenhaixiang (A)
Quoted 38 lines
Quoted 2 lines

I'm sorry for the delay. Here are some details from the boot log and

/proc/iomem:

Quoted 2 lines

The Boot log:

[    0.000000] Linux version 6.8.0 ([email protected]) (gcc (GCC)

10.3.1, GNU ld (GNU Binutils) 2.37) #3 SMP PREEMPT_DYNAMIC Wed Mar 20 11:46:11 UTC 2024

Quoted 1 line
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-6.8.0

root=/dev/mapper/root ro crashkernel=512M resume=/dev/mapper/swap rd.lvm.lv=root rd.lvm.lv=swap crash_kexec_post_notifiers softlockup_panic=1 reserve_kbox_mem=16M fsck.mode=auto fsck.repair=yes panic=3 nmi_watchdog=1 quiet rd.shell=0 memblock=debug efi=debug console=ttyS0,115200n8 console=tty0
......snip...

Quoted 1 line
[    0.022622] memblock_phys_alloc_range: 536870912 bytes align=0x1000000

from=0x0000000000000000 max_addr=0x0000000100000000 reserve_crashkernel_generic+0x7c/0x220

Quoted 1 line
[    0.022628] memblock_phys_alloc_range: 536870912 bytes align=0x1000000

from=0x0000000100000000 max_addr=0x0000400000000000 reserve_crashkernel_generic+0x7c/0x220

Quoted 1 line
[    0.022632] memblock_reserve: [0x000000c01f000000-0x000000c03effffff]

memblock_alloc_range_nid+0xee/0x170

Quoted 1 line
[    0.022634] memblock_phys_alloc_range: 268435456 bytes align=0x1000000

from=0x0000000000000000 max_addr=0x0000000100000000 reserve_crashkernel_generic+0x11d/0x220

Quoted 1 line
[    0.022638] memblock_reserve: [0x0000000049000000-0x0000000058ffffff]

memblock_alloc_range_nid+0xee/0x170

Quoted 1 line
[    0.022640] crashkernel low memory reserved: 0x49000000 - 0x59000000

(256 MB)

Quoted 1 line
[    0.022641] crashkernel reserved: 0x000000c01f000000 -

0x000000c03f000000 (512 MB)

Here, crashkernel,low is reserved in region: [0x49000000 - 0x59000000] (256 MB)

      crashkernel,high is reserved in region: [0x000000c01f000000 -

0x000000c03f000000] (512 MB) ......

Quoted 1 line
[    0.029839] memblock_reserve: [0x000000c03ffff740-0x000000c03fffff7f]

memblock_alloc_range_nid+0xee/0x170

Quoted 8 lines
Quoted 8 lines
Quoted 1 line
[    0.029843] e820: update [mem 0x53cbd000-0x53ccffff] usable ==>

reserved

Quoted 1 line
[    0.029861] TSC deadline timer available

Then here, region [0x53cbd000-0x53ccffff] is reserved in e820, and print abvoe "usable ==> reserved". This should be the step which prevents earlier reserved crashkernel,low from being added to iomem tree. I am not sure what triggered the e820 update.

We added dump_stack () printing in efi_mem_reserve () and found that [0x53cbd000-0x53ccffff] was reserved by BGRT:

  [    0.032259] e820: update [mem 0x53cbd000-0x53ccffff] usable ==>
reserved
  [    0.032262] CPU: 0 PID: 0 Comm: swapper Not tainted
5.10.0-60.18.0.50.h820.eulerosv2r11.x86_64 #7
  [    0.032263] Hardware name: Huawei 2288H V5/BC11SPSCB0, BIOS 8.25
08/30/2022
  [    0.032264] Call Trace:
  [    0.032265]  ? dump_stack+0x57/0x6e
  [    0.032267]  ? bgrt_init+0xc2/0xc2
  [    0.032268]  ? __e820__range_update+0x7a/0x1d6
  [    0.032270]  ? bgrt_init+0xc2/0xc2
  [    0.032272]  ? bgrt_init+0xc2/0xc2
  [    0.032274]  ? efi_arch_mem_reserve+0x1a3/0x1d0
  [    0.032276]  ? efi_mem_reserve+0x2d/0x42
  [    0.032278]  ? acpi_parse_bgrt+0xa/0x11
  [    0.032279]  ? acpi_table_parse+0x86/0xbc
  [    0.032281]  ? acpi_boot_init+0x79/0xad
  [    0.032282]  ? setup_arch+0x835/0x954
  [    0.032284]  ? start_kernel+0x5d/0x455
  [    0.032286]  ? secondary_startup_64_no_verify+0xc2/0xcb

efi_reserve_boot_services() has reserved memory of type EFI_BOOT_SERVICES_CODE & EFI_BOOT_SERVICES_DATA before crashkernel. efi_bgrt_init() assumes that EFI_BOOT_SERVICES_DATA is not reserved by other modules. Then, the e820_table is directly updated, and the BGRT memory is reserved.

However, memblock_is_region_reserved() in efi_reserve_boot_services() returns true when the ranges only overlap.

Quoted 6 lines
Quoted 1 line
     already_reserved = memblock_is_region_reserved(start, size);

Do you mean efi_reserve_boot_services is supposed to reserve the bgrt memory but it does not reserve it due to the region overlapping with some other reserved region? If so can you debug and find what exact memblock reserved region overlaps with the bgrt?

Yes. I added the following debug print to efi_reserve_boot_services():

arch/x86/platform/efi/quirks.c void __init efi_reserve_boot_services(void) · line 348

Quoted 10 lines
arch/x86/platform/efi/quirks.c +4 -0
@@ -339,6 +339,10 @@ void __init efi_reserve_boot_services(void)
 
                already_reserved = memblock_is_region_reserved(start, size);
 
+               pr_info("kdumpdebug: efi_reserve_boot_services start 0x%lu, "
+                       "size 0x%lx, type 0x%lx, already_reserved %d\n",
+                       start, size, md->type, already_reserved);
+
                /*
                 * Because the following memblock_reserve() is paired
                 * with memblock_free_late() for this region in

It's great debugging and analysis, thanks you guys. Now there are several questions:

Quoted 3 lines

1) why memory region [0x5976a018-0x5976abc7] is reserved by memblock for efi_mem_attr_table. It's supposed to be outside of the EFI_BOOT_SERVICES_DATA area? We may need check here if it's a bug.

The mem_attr_table memory falls into a EFI Boot Service Data region

[    0.000000] efi: mem22: [Boot Data   |   |  |  |  |  |  |  |  |  |
 |WB|WT|WC|UC] range=[0x0000000051329000-0x000000005cefefff] (187MB)
Quoted 9 lines
[    0.000000] random: crng init done
[    0.000000] memblock_reserve: [0x000000005976a018-0x000000005976abc7] efi_memattr_init+0x51/0xa0
Quoted 5 lines

This memory [0x0000005976a018-0x00000005976abc7] is reserved here, which belongs to EFI_BOOT_SERVICES_DATA.

    [    0.000000] memblock_reserve: [0x000000005976a018-0x000000005976abc7] efi_memattr_init+0x51/0xa0
It falls in the following range
    [    0.000000] efi: mem22: [Boot Data   |   |  |  |  |  |  |  |  |  |   |WB|WT|WC|UC] range=[0x0000000051329000-0x000000005cefefff] (187MB)
Quoted 8 lines
Quoted 2 lines

in efi_reserve_boot_services(), [0x0000005132900-0x00000005cefeff] will not be fully reserved because [0x0000005976a018-0x0000005976abc7] has already been reserved and overlaps with [0x0000005976a018-0x0000005976abc7]

2) Because efi_mem_attr_table memblock reserved [0x5976a018-0x5976abc7], the whole EFI_BOOT_SERVICES_DATA area [0x5132900-0x5cefeff] is not memblock reserved for later free. Excep of the small area, do we need still memblock reserve the remaining area, we may need check if this is a bug.

I think the whole EFI Boot Data region should be reserved temperately by efi_reserve_boot_services, but if they should be reserved partially as multiple smaller regions I'm not sure, I added Ard and EFI list in another reply, let's see how EFI people think.

Quoted 9 lines
Quoted 9 lines

    [    0.021316] efi: kdumpdebug: efi_reserve_boot_services start 0x51329000, size 0xbbd6000, type 0x4, already_reserved 1

It is not reserved by memblock, this free memory region is allocated by crashkernel

    [    0.022597] crashkernel low memory reserved: 0x49000000 - 0x59000000 (256 MB)
    [    0.022599] crashkernel reserved: 0x000000c01f000000 - 0x000000c03f000000 (512 MB)
Quoted 6 lines
Quoted 3 lines

In efi_bgrt_init (), it is assumed that the memory of the EFI_BOOT_SERVICES_DATA type has been successfully reserved. Therefore, the address in the range is directly used. As a result, the memory overlaps with the crashkernel region.

(3) efi_bgrt_init() should be innocent because it's supposed to safely use the area according to the existing efi quirk handling.

Agreed

Quoted 9 lines

(4) the deferring of adding crashh_low_res to iomem exposed the above efi issue. When we cancel the deferring of crashh_res inserting into iomem, we can see that the brgt area is reserved inside crashkernel region, that's problematic.

2d4fd058-60efefff : System RAM
2d4fd058-58ffffff : System RAM

Quoted 94 lines
    49000000-58ffffff : Crash kernel
      53cbd000-53ccffff : Reserved     <---

60eff000-704fefff : Reserved
--
93dd424000-93dd9fffff : Kernel bss
c01f000000-c03effffff : Crash kernel
d0000000000-d0fffffffff : PCI Bus 0000:00
d0000000000-d00001fffff : PCI Bus 0000:01

Quoted 84 lines
    [    0.029694] e820: update [mem 0x53cbd000-0x53ccffff] usable ==> reserved
Quoted 3 lines

BTW, the previous email threads are weird, and not threading correctly, hard to find information.

It should be because the log content is too large and has been put on hold. In my previous email, I received a prompt:

The reason it is being held:

Message body is too big: 248998 bytes with a limit of 40 KB

Quoted 70 lines
Quoted 60 lines

/*
* Because the following memblock_reserve() is paired
* with memblock_free_late() for this region in
* efi_free_boot_services(), we must be extremely
* careful not to reserve, and subsequently free,
* critical regions of memory (like the kernel image) or
* those regions that somebody else has already
* reserved.
*
* A good example of a critical region that must not be
* freed is page zero (first 4Kb of memory), which may
* contain boot services code/data but is marked
* E820_TYPE_RESERVED by trim_bios_range().
*/
if (!already_reserved) {
memblock_reserve(start, size);

/*
* If we are the first to reserve the region, no
* one else cares about it. We own it and can
* free it later.
*/
if (can_free_region(start, size))
continue;
}

As a result, some memory of EFI_BOOT_SERVICES_DATA is not reserved in advance. The subsequent crashkernel happens to reserve this portion of memory, which conflicts with BGRT.

Quoted 28 lines

Current analysis suggests that efi_reserve_boot_services() is causing the update of the e820 table.

Quoted 3 lines

How do you boot into your new 6.8.0 kernel? Used kexec -l to jump into the 2nd kernel, or reboot from bios/firmware boot up into 6.8.0?

It's reboot from bios boot up into 6.8.0. I attempted to revert the below patch,
and this time the conflicting segment "53cbd000-53ccffff" also appeared in the /proc/iomem of the 6.8 kernel.

2d4fd058-60efefff : System RAM
2d4fd058-58ffffff : System RAM

    49000000-58ffffff : Crash kernel
      53cbd000-53ccffff : Reserved

60eff000-704fefff : Reserved
--
93dd424000-93dd9fffff : Kernel bss
c01f000000-c03effffff : Crash kernel
d0000000000-d0fffffffff : PCI Bus 0000:00
d0000000000-d00001fffff : PCI Bus 0000:01

Quoted 8 lines

Reverting below commit should fix your problem, can you try it?

commit 4a693ce65b186fddc1a73621bd6f941e6e3eca21

Author: Huacai Chen <[email protected]>
Date:   Fri Dec 29 16:02:13 2023 +0800

kdump: defer the insertion of crashkernel resources

Quoted 7 lines
Quoted 2 lines

.

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

.

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#6 lore raw
In reply to Li Huafei
Quoted 114 lines · Li Huafei
Quoted 87 lines · Dave Young

Hi,

Quoted 84 lines · Li Huafei

Hi Baoquan,

Quoted 38 lines · chenhaixiang (A)
Quoted 38 lines
Quoted 2 lines

I'm sorry for the delay. Here are some details from the boot log and

/proc/iomem:

Quoted 2 lines

The Boot log:

[    0.000000] Linux version 6.8.0 ([email protected]) (gcc (GCC)

10.3.1, GNU ld (GNU Binutils) 2.37) #3 SMP PREEMPT_DYNAMIC Wed Mar 20 11:46:11 UTC 2024

Quoted 1 line
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-6.8.0

root=/dev/mapper/root ro crashkernel=512M resume=/dev/mapper/swap rd.lvm.lv=root rd.lvm.lv=swap crash_kexec_post_notifiers softlockup_panic=1 reserve_kbox_mem=16M fsck.mode=auto fsck.repair=yes panic=3 nmi_watchdog=1 quiet rd.shell=0 memblock=debug efi=debug console=ttyS0,115200n8 console=tty0
......snip...

Quoted 1 line
[    0.022622] memblock_phys_alloc_range: 536870912 bytes align=0x1000000

from=0x0000000000000000 max_addr=0x0000000100000000 reserve_crashkernel_generic+0x7c/0x220

Quoted 1 line
[    0.022628] memblock_phys_alloc_range: 536870912 bytes align=0x1000000

from=0x0000000100000000 max_addr=0x0000400000000000 reserve_crashkernel_generic+0x7c/0x220

Quoted 1 line
[    0.022632] memblock_reserve: [0x000000c01f000000-0x000000c03effffff]

memblock_alloc_range_nid+0xee/0x170

Quoted 1 line
[    0.022634] memblock_phys_alloc_range: 268435456 bytes align=0x1000000

from=0x0000000000000000 max_addr=0x0000000100000000 reserve_crashkernel_generic+0x11d/0x220

Quoted 1 line
[    0.022638] memblock_reserve: [0x0000000049000000-0x0000000058ffffff]

memblock_alloc_range_nid+0xee/0x170

Quoted 1 line
[    0.022640] crashkernel low memory reserved: 0x49000000 - 0x59000000

(256 MB)

Quoted 1 line
[    0.022641] crashkernel reserved: 0x000000c01f000000 -

0x000000c03f000000 (512 MB)

Here, crashkernel,low is reserved in region: [0x49000000 - 0x59000000] (256 MB)

      crashkernel,high is reserved in region: [0x000000c01f000000 -

0x000000c03f000000] (512 MB) ......

Quoted 1 line
[    0.029839] memblock_reserve: [0x000000c03ffff740-0x000000c03fffff7f]

memblock_alloc_range_nid+0xee/0x170

Quoted 8 lines
Quoted 8 lines
Quoted 1 line
[    0.029843] e820: update [mem 0x53cbd000-0x53ccffff] usable ==>

reserved

Quoted 1 line
[    0.029861] TSC deadline timer available

Then here, region [0x53cbd000-0x53ccffff] is reserved in e820, and print abvoe "usable ==> reserved". This should be the step which prevents earlier reserved crashkernel,low from being added to iomem tree. I am not sure what triggered the e820 update.

We added dump_stack () printing in efi_mem_reserve () and found that [0x53cbd000-0x53ccffff] was reserved by BGRT:

  [    0.032259] e820: update [mem 0x53cbd000-0x53ccffff] usable ==>
reserved
  [    0.032262] CPU: 0 PID: 0 Comm: swapper Not tainted
5.10.0-60.18.0.50.h820.eulerosv2r11.x86_64 #7
  [    0.032263] Hardware name: Huawei 2288H V5/BC11SPSCB0, BIOS 8.25
08/30/2022
  [    0.032264] Call Trace:
  [    0.032265]  ? dump_stack+0x57/0x6e
  [    0.032267]  ? bgrt_init+0xc2/0xc2
  [    0.032268]  ? __e820__range_update+0x7a/0x1d6
  [    0.032270]  ? bgrt_init+0xc2/0xc2
  [    0.032272]  ? bgrt_init+0xc2/0xc2
  [    0.032274]  ? efi_arch_mem_reserve+0x1a3/0x1d0
  [    0.032276]  ? efi_mem_reserve+0x2d/0x42
  [    0.032278]  ? acpi_parse_bgrt+0xa/0x11
  [    0.032279]  ? acpi_table_parse+0x86/0xbc
  [    0.032281]  ? acpi_boot_init+0x79/0xad
  [    0.032282]  ? setup_arch+0x835/0x954
  [    0.032284]  ? start_kernel+0x5d/0x455
  [    0.032286]  ? secondary_startup_64_no_verify+0xc2/0xcb

efi_reserve_boot_services() has reserved memory of type EFI_BOOT_SERVICES_CODE & EFI_BOOT_SERVICES_DATA before crashkernel. efi_bgrt_init() assumes that EFI_BOOT_SERVICES_DATA is not reserved by other modules. Then, the e820_table is directly updated, and the BGRT memory is reserved.

However, memblock_is_region_reserved() in efi_reserve_boot_services() returns true when the ranges only overlap.

Quoted 6 lines
Quoted 1 line
     already_reserved = memblock_is_region_reserved(start, size);

Do you mean efi_reserve_boot_services is supposed to reserve the bgrt memory but it does not reserve it due to the region overlapping with some other reserved region? If so can you debug and find what exact memblock reserved region overlaps with the bgrt?

Yes. I added the following debug print to efi_reserve_boot_services():

arch/x86/platform/efi/quirks.c +4 -0
@@ -339,6 +339,10 @@ void __init efi_reserve_boot_services(void)
 
                already_reserved = memblock_is_region_reserved(start, size);
 
+               pr_info("kdumpdebug: efi_reserve_boot_services start 0x%lu, "
+                       "size 0x%lx, type 0x%lx, already_reserved %d\n",
+                       start, size, md->type, already_reserved);
+
                /*
                 * Because the following memblock_reserve() is paired
                 * with memblock_free_late() for this region in
Quoted 7 lines

This memory [0x0000005976a018-0x00000005976abc7] is reserved here, which belongs to EFI_BOOT_SERVICES_DATA.

    [    0.000000] memblock_reserve: [0x000000005976a018-0x000000005976abc7] efi_memattr_init+0x51/0xa0
It falls in the following range
    [    0.000000] efi: mem22: [Boot Data   |   |  |  |  |  |  |  |  |  |   |WB|WT|WC|UC] range=[0x0000000051329000-0x000000005cefefff] (187MB)

in efi_reserve_boot_services(), [0x0000005132900-0x00000005cefeff] will not be fully reserved because [0x0000005976a018-0x0000005976abc7] has already been reserved and overlaps with [0x0000005976a018-0x0000005976abc7]

Ok, it looks to me it is like this:

efi_memattr_init() reserved the memattr table with memblock_reserve efi_reserve_boot_services failed to reserve the boot data region which includes the memattr table due to it has been revervely partially.

So this should be a UEFI issue revealed by the crashkernel resource
late intert commit. I suspect the memblock_reserve in
efi_memattr_init can be just removed and leave to efi_reserve_boot_services to do that.

Added Ard and efi list for opinion.

Quoted 5 lines
    [    0.021316] efi: kdumpdebug: efi_reserve_boot_services start 0x51329000, size 0xbbd6000, type 0x4, already_reserved 1

It is not reserved by memblock, this free memory region is allocated by crashkernel

Quoted 8 lines
    [    0.022597] crashkernel low memory reserved: 0x49000000 - 0x59000000 (256 MB)
    [    0.022599] crashkernel reserved: 0x000000c01f000000 - 0x000000c03f000000 (512 MB)

In efi_bgrt_init (), it is assumed that the memory of the EFI_BOOT_SERVICES_DATA type has been successfully reserved. Therefore, the address in the range is directly used. As a result, the memory overlaps with the crashkernel region.

    [    0.029694] e820: update [mem 0x53cbd000-0x53ccffff] usable ==> reserved

Thanks
Dave

#7 lore raw
Quoted 120 lines · Dave Young
Quoted 119 lines · Li Huafei
Quoted 87 lines · Dave Young

Hi,

Quoted 84 lines · Li Huafei

Hi Baoquan,

Quoted 38 lines · chenhaixiang (A)
Quoted 38 lines
Quoted 2 lines

I'm sorry for the delay. Here are some details from the boot log and

/proc/iomem:

Quoted 2 lines

The Boot log:

[    0.000000] Linux version 6.8.0 ([email protected]) (gcc (GCC)

10.3.1, GNU ld (GNU Binutils) 2.37) #3 SMP PREEMPT_DYNAMIC Wed Mar 20 11:46:11 UTC 2024

Quoted 1 line
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-6.8.0

root=/dev/mapper/root ro crashkernel=512M resume=/dev/mapper/swap rd.lvm.lv=root rd.lvm.lv=swap crash_kexec_post_notifiers softlockup_panic=1 reserve_kbox_mem=16M fsck.mode=auto fsck.repair=yes panic=3 nmi_watchdog=1 quiet rd.shell=0 memblock=debug efi=debug console=ttyS0,115200n8 console=tty0
......snip...

Quoted 1 line
[    0.022622] memblock_phys_alloc_range: 536870912 bytes align=0x1000000

from=0x0000000000000000 max_addr=0x0000000100000000 reserve_crashkernel_generic+0x7c/0x220

Quoted 1 line
[    0.022628] memblock_phys_alloc_range: 536870912 bytes align=0x1000000

from=0x0000000100000000 max_addr=0x0000400000000000 reserve_crashkernel_generic+0x7c/0x220

Quoted 1 line
[    0.022632] memblock_reserve: [0x000000c01f000000-0x000000c03effffff]

memblock_alloc_range_nid+0xee/0x170

Quoted 1 line
[    0.022634] memblock_phys_alloc_range: 268435456 bytes align=0x1000000

from=0x0000000000000000 max_addr=0x0000000100000000 reserve_crashkernel_generic+0x11d/0x220

Quoted 1 line
[    0.022638] memblock_reserve: [0x0000000049000000-0x0000000058ffffff]

memblock_alloc_range_nid+0xee/0x170

Quoted 1 line
[    0.022640] crashkernel low memory reserved: 0x49000000 - 0x59000000

(256 MB)

Quoted 1 line
[    0.022641] crashkernel reserved: 0x000000c01f000000 -

0x000000c03f000000 (512 MB)

Here, crashkernel,low is reserved in region: [0x49000000 - 0x59000000] (256 MB)

      crashkernel,high is reserved in region: [0x000000c01f000000 -

0x000000c03f000000] (512 MB) ......

Quoted 1 line
[    0.029839] memblock_reserve: [0x000000c03ffff740-0x000000c03fffff7f]

memblock_alloc_range_nid+0xee/0x170

Quoted 8 lines
Quoted 8 lines
Quoted 1 line
[    0.029843] e820: update [mem 0x53cbd000-0x53ccffff] usable ==>

reserved

Quoted 1 line
[    0.029861] TSC deadline timer available

Then here, region [0x53cbd000-0x53ccffff] is reserved in e820, and print abvoe "usable ==> reserved". This should be the step which prevents earlier reserved crashkernel,low from being added to iomem tree. I am not sure what triggered the e820 update.

We added dump_stack () printing in efi_mem_reserve () and found that [0x53cbd000-0x53ccffff] was reserved by BGRT:

  [    0.032259] e820: update [mem 0x53cbd000-0x53ccffff] usable ==>
reserved
  [    0.032262] CPU: 0 PID: 0 Comm: swapper Not tainted
5.10.0-60.18.0.50.h820.eulerosv2r11.x86_64 #7
  [    0.032263] Hardware name: Huawei 2288H V5/BC11SPSCB0, BIOS 8.25
08/30/2022
  [    0.032264] Call Trace:
  [    0.032265]  ? dump_stack+0x57/0x6e
  [    0.032267]  ? bgrt_init+0xc2/0xc2
  [    0.032268]  ? __e820__range_update+0x7a/0x1d6
  [    0.032270]  ? bgrt_init+0xc2/0xc2
  [    0.032272]  ? bgrt_init+0xc2/0xc2
  [    0.032274]  ? efi_arch_mem_reserve+0x1a3/0x1d0
  [    0.032276]  ? efi_mem_reserve+0x2d/0x42
  [    0.032278]  ? acpi_parse_bgrt+0xa/0x11
  [    0.032279]  ? acpi_table_parse+0x86/0xbc
  [    0.032281]  ? acpi_boot_init+0x79/0xad
  [    0.032282]  ? setup_arch+0x835/0x954
  [    0.032284]  ? start_kernel+0x5d/0x455
  [    0.032286]  ? secondary_startup_64_no_verify+0xc2/0xcb

efi_reserve_boot_services() has reserved memory of type EFI_BOOT_SERVICES_CODE & EFI_BOOT_SERVICES_DATA before crashkernel. efi_bgrt_init() assumes that EFI_BOOT_SERVICES_DATA is not reserved by other modules. Then, the e820_table is directly updated, and the BGRT memory is reserved.

However, memblock_is_region_reserved() in efi_reserve_boot_services() returns true when the ranges only overlap.

Quoted 6 lines
Quoted 1 line
     already_reserved = memblock_is_region_reserved(start, size);

Do you mean efi_reserve_boot_services is supposed to reserve the bgrt memory but it does not reserve it due to the region overlapping with some other reserved region? If so can you debug and find what exact memblock reserved region overlaps with the bgrt?

Yes. I added the following debug print to efi_reserve_boot_services():

arch/x86/platform/efi/quirks.c +4 -0
@@ -339,6 +339,10 @@ void __init efi_reserve_boot_services(void)
 
                already_reserved = memblock_is_region_reserved(start, size);
 
+               pr_info("kdumpdebug: efi_reserve_boot_services start 0x%lu, "
+                       "size 0x%lx, type 0x%lx, already_reserved %d\n",
+                       start, size, md->type, already_reserved);
+
                /*
                 * Because the following memblock_reserve() is paired
                 * with memblock_free_late() for this region in

This memory [0x0000005976a018-0x00000005976abc7] is reserved here, which belongs to EFI_BOOT_SERVICES_DATA.

    [    0.000000] memblock_reserve: [0x000000005976a018-0x000000005976abc7] efi_memattr_init+0x51/0xa0
It falls in the following range
    [    0.000000] efi: mem22: [Boot Data   |   |  |  |  |  |  |  |  |  |   |WB|WT|WC|UC] range=[0x0000000051329000-0x000000005cefefff] (187MB)
Quoted 9 lines
Quoted 2 lines

in efi_reserve_boot_services(), [0x0000005132900-0x00000005cefeff] will not be fully reserved because [0x0000005976a018-0x0000005976abc7] has already been reserved and overlaps with [0x0000005976a018-0x0000005976abc7]

Ok, it looks to me it is like this:

efi_memattr_init() reserved the memattr table with memblock_reserve efi_reserve_boot_services failed to reserve the boot data region which includes the memattr table due to it has been revervely partially.

Yes, that's my analysis.

Quoted 7 lines

So this should be a UEFI issue revealed by the crashkernel resource
late intert commit. I suspect the memblock_reserve in
efi_memattr_init can be just removed and leave to efi_reserve_boot_services to do that.

Added Ard and efi list for opinion.

Thanks a lot! I was just about to ask the efi guys for help.

Quoted 20 lines
Quoted 6 lines

    [    0.021316] efi: kdumpdebug: efi_reserve_boot_services start 0x51329000, size 0xbbd6000, type 0x4, already_reserved 1

It is not reserved by memblock, this free memory region is allocated by crashkernel

Quoted 8 lines
    [    0.022597] crashkernel low memory reserved: 0x49000000 - 0x59000000 (256 MB)
    [    0.022599] crashkernel reserved: 0x000000c01f000000 - 0x000000c03f000000 (512 MB)

In efi_bgrt_init (), it is assumed that the memory of the EFI_BOOT_SERVICES_DATA type has been successfully reserved. Therefore, the address in the range is directly used. As a result, the memory overlaps with the crashkernel region.

    [    0.029694] e820: update [mem 0x53cbd000-0x53ccffff] usable ==> reserved

Thanks
Dave

.