Skip to contentSkip to conversation

riscv: Properly export reserved regions in /proc/iomem

PATCH Applied Björn Töpel 5 participants 14 messages Apr 9, 2025 – Jul 16, 2025 RISC-V

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 riscv: Properly export reserved regions in /proc/iomem

From: Björn Töpel <[email protected]>

The /proc/iomem represents the kernel's memory map. Regions marked with "Reserved" tells the user that the range should not be tampered with. Kexec-tools, when using the older kexec_load syscall relies on the "Reserved" regions to build the memory segments, that will be the target of the new kexec'd kernel.

The RISC-V port tries to expose all reserved regions to userland, but some regions were not properly exposed: Regions that resided in both the "regular" and reserved memory block, e.g. the EFI Memory Map. A missing entry could result in reserved memory being overwritten.

It turns out, that arm64, and loongarch had a similar issue a while back:

commit d91680e687f4 ("arm64: Fix /proc/iomem for reserved but not memory regions") commit 50d7ba36b916 ("arm64: export memblock_reserve()d regions via /proc/iomem")

Similar to the other ports, resolve the issue by splitting the regions in an arch initcall, since we need a working allocator.

Fixes ffe0e5261268 ("RISC-V: Improve init_resources()")
---
 arch/riscv/kernel/setup.c | 36 +++++++++++++++++++++++++++++++++++-
 1 file changed, 35 insertions(+), 1 deletion(-)
arch/riscv/kernel/setup.c +35 -1
index c174544eefc8..f7c9a1caa83e 100644
@@ -66,6 +66,9 @@ static struct resource bss_res = { .name = "Kernel bss", };
 static struct resource elfcorehdr_res = { .name = "ELF Core hdr", };
 #endif
 
+static int num_standard_resources;
+static struct resource *standard_resources;
+
 static int __init add_resource(struct resource *parent,
 				struct resource *res)
 {
@@ -139,7 +142,7 @@ static void __init init_resources(void)
 	struct resource *res = NULL;
 	struct resource *mem_res = NULL;
 	size_t mem_res_sz = 0;
-	int num_resources = 0, res_idx = 0;
+	int num_resources = 0, res_idx = 0, non_resv_res = 0;
 	int ret = 0;
 
 	/* + 1 as memblock_alloc() might increase memblock.reserved.cnt */
@@ -193,6 +196,7 @@ static void __init init_resources(void)
 	/* Add /memory regions to the resource tree */
 	for_each_mem_region(region) {
 		res = &mem_res[res_idx--];
+		non_resv_res++;
 
 		if (unlikely(memblock_is_nomap(region))) {
 			res->name = "Reserved";
@@ -210,6 +214,9 @@ static void __init init_resources(void)
 			goto error;
 	}
 
+	num_standard_resources = non_resv_res;
+	standard_resources = &mem_res[res_idx + 1];
+
 	/* Clean-up any unused pre-allocated resources */
 	if (res_idx >= 0)
 		memblock_free(mem_res, (res_idx + 1) * sizeof(*mem_res));
@@ -221,6 +228,33 @@ static void __init init_resources(void)
 	memblock_free(mem_res, mem_res_sz);
 }
 
+static int __init reserve_memblock_reserved_regions(void)
+{
+	u64 i, j;
+
+	for (i = 0; i < num_standard_resources; i++) {
+		struct resource *mem = &standard_resources[i];
+		phys_addr_t r_start, r_end, mem_size = resource_size(mem);
+
+		if (!memblock_is_region_reserved(mem->start, mem_size))
+			continue;
+
+		for_each_reserved_mem_range(j, &r_start, &r_end) {
+			resource_size_t start, end;
+
+			start = max(PFN_PHYS(PFN_DOWN(r_start)), mem->start);
+			end = min(PFN_PHYS(PFN_UP(r_end)) - 1, mem->end);
+
+			if (start > mem->end || end < mem->start)
+				continue;
+
+			reserve_region_with_split(mem, start, end, "Reserved");
+		}
+	}
+
+	return 0;
+}
+arch_initcall(reserve_memblock_reserved_regions);
 
 static void __init parse_dtb(void)
 {

base-commit: a24588245776dafc227243a01bfbeb8a59bafba9

Signature · 1 line
2.45.2
reviewer Reviewed-by on v1 1/1
#1 lore raw

Hi Bjorn,

Quoted 104 lines · Björn Töpel

From: Björn Töpel <[email protected]>

The /proc/iomem represents the kernel's memory map. Regions marked with "Reserved" tells the user that the range should not be tampered with. Kexec-tools, when using the older kexec_load syscall relies on the "Reserved" regions to build the memory segments, that will be the target of the new kexec'd kernel.

The RISC-V port tries to expose all reserved regions to userland, but some regions were not properly exposed: Regions that resided in both the "regular" and reserved memory block, e.g. the EFI Memory Map. A missing entry could result in reserved memory being overwritten.

It turns out, that arm64, and loongarch had a similar issue a while back:

commit d91680e687f4 ("arm64: Fix /proc/iomem for reserved but not memory regions") commit 50d7ba36b916 ("arm64: export memblock_reserve()d regions via /proc/iomem")

Similar to the other ports, resolve the issue by splitting the regions in an arch initcall, since we need a working allocator.

Fixes ffe0e5261268 ("RISC-V: Improve init_resources()")
---
  arch/riscv/kernel/setup.c | 36 +++++++++++++++++++++++++++++++++++-
  1 file changed, 35 insertions(+), 1 deletion(-)
arch/riscv/kernel/setup.c +35 -1
index c174544eefc8..f7c9a1caa83e 100644
@@ -66,6 +66,9 @@ static struct resource bss_res = { .name = "Kernel bss", };
  static struct resource elfcorehdr_res = { .name = "ELF Core hdr", };
  #endif
 
+static int num_standard_resources;
+static struct resource *standard_resources;
+
  static int __init add_resource(struct resource *parent,
  				struct resource *res)
  {
@@ -139,7 +142,7 @@ static void __init init_resources(void)
  	struct resource *res = NULL;
  	struct resource *mem_res = NULL;
  	size_t mem_res_sz = 0;
-	int num_resources = 0, res_idx = 0;
+	int num_resources = 0, res_idx = 0, non_resv_res = 0;
  	int ret = 0;
 
  	/* + 1 as memblock_alloc() might increase memblock.reserved.cnt */
@@ -193,6 +196,7 @@ static void __init init_resources(void)
  	/* Add /memory regions to the resource tree */
  	for_each_mem_region(region) {
  		res = &mem_res[res_idx--];
+		non_resv_res++;
 
  		if (unlikely(memblock_is_nomap(region))) {
  			res->name = "Reserved";
@@ -210,6 +214,9 @@ static void __init init_resources(void)
  			goto error;
  	}
 
+	num_standard_resources = non_resv_res;
+	standard_resources = &mem_res[res_idx + 1];
+
  	/* Clean-up any unused pre-allocated resources */
  	if (res_idx >= 0)
  		memblock_free(mem_res, (res_idx + 1) * sizeof(*mem_res));
@@ -221,6 +228,33 @@ static void __init init_resources(void)
  	memblock_free(mem_res, mem_res_sz);
  }
 
+static int __init reserve_memblock_reserved_regions(void)
+{
+	u64 i, j;
+
+	for (i = 0; i < num_standard_resources; i++) {
+		struct resource *mem = &standard_resources[i];
+		phys_addr_t r_start, r_end, mem_size = resource_size(mem);
+
+		if (!memblock_is_region_reserved(mem->start, mem_size))
+			continue;
+
+		for_each_reserved_mem_range(j, &r_start, &r_end) {
+			resource_size_t start, end;
+
+			start = max(PFN_PHYS(PFN_DOWN(r_start)), mem->start);
+			end = min(PFN_PHYS(PFN_UP(r_end)) - 1, mem->end);
+
+			if (start > mem->end || end < mem->start)
+				continue;
+
+			reserve_region_with_split(mem, start, end, "Reserved");
+		}
+	}
+
+	return 0;
+}
+arch_initcall(reserve_memblock_reserved_regions);
 
  static void __init parse_dtb(void)
  {
Quoted 1 line

base-commit: a24588245776dafc227243a01bfbeb8a59bafba9

It will be merged with other fixes in the next rc-ish!

Thanks,

Alex

#2 lore raw
In reply to Björn Töpel
This patch was applied to riscv/linux.git (fixes) by Alexandre Ghiti <[email protected]>:

Hello:

This patch was applied to riscv/linux.git (fixes)
by Alexandre Ghiti <[email protected]>:

Quoted 9 lines · On Wed, 9 Apr 2025 20:21:27 +0200 you wrote:

From: Björn Töpel <[email protected]>

The /proc/iomem represents the kernel's memory map. Regions marked with "Reserved" tells the user that the range should not be tampered with. Kexec-tools, when using the older kexec_load syscall relies on the "Reserved" regions to build the memory segments, that will be the target of the new kexec'd kernel.

[...]

Here is the summary with links:
- [fixes] riscv: Properly export reserved regions in /proc/iomem

    https://git.kernel.org/riscv/c/e94eb7ea6f20

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
#3 lore raw

cover Fix for riscv vmcore issue

In reply to Björn Töpel

We are creating a vmcore using kexec on a Linux 6.15 RISC-V system and analyzing it with the crash tool on the host. This workflow used to work on Linux 6.14 but is now broken in 6.15.

The issue is caused by a change in the kernel:
In Linux 6.15, certain memblock sections are now marked as Reserved in /proc/iomem.
The kexec tool excludes all Reserved regions when generating the vmcore, so these sections are missing from the dump.

However, the kernel still uses addresses in these regions—for example, for IRQ pointers.
Since the crash tool needs access to these memory areas to function correctly, their exclusion breaks the analysis.

Pnina Feder (1):
riscv: Change memblock reserved name to be recognized on kexec

 arch/riscv/kernel/setup.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Signature · 1 line
2.43.0
#13 lore raw

PATCH 1/1 riscv: Change memblock reserved name to be recognized on kexec

In reply to Björn Töpel

memblock resreved exposes as Reserved on iomem,
kexec tool doesn't take those parts to vmcore, but
the kernel use those address and it needed when opening the vmcore. Without this fix the crash-tool fails.

Fixes e94eb7ea6f20 ("riscv: Properly export reserved regions in /proc/iomem")
---
 arch/riscv/kernel/setup.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/riscv/kernel/setup.c +1 -1
index 14888e5ea19a..16e0cebed170 100644
@@ -249,7 +249,7 @@ static int __init reserve_memblock_reserved_regions(void)
 			if (start > mem->end || end < mem->start)
 				continue;
 
-			reserve_region_with_split(mem, start, end, "Reserved");
+			reserve_region_with_split(mem, start, end, "Reserved-memblock");
 		}
 	}
 
Signature · 1 line
2.43.0
#4 lore raw

Re: [PATCH 0/1] Fix for riscv vmcore issue

In reply to Pnina Feder

Pnina!

Quoted 3 lines · Pnina Feder

We are creating a vmcore using kexec on a Linux 6.15 RISC-V system and analyzing it with the crash tool on the host. This workflow used to work on Linux 6.14 but is now broken in 6.15.

Thanks for reporting this!

Quoted 4 lines

The issue is caused by a change in the kernel:
In Linux 6.15, certain memblock sections are now marked as Reserved in /proc/iomem. The kexec tool excludes all Reserved regions when generating the vmcore, so these sections are missing from the dump.

How are you collecting the /proc/vmcore file? A full set of commands would be helpful.

Quoted 3 lines

However, the kernel still uses addresses in these regions—for example, for IRQ pointers. Since the crash tool needs access to these memory areas to function correctly, their exclusion breaks the analysis.

Wdym with "IRQ pointers"? Also, what version (sha1) of crash are you using?

Thanks!
Björn

#5 lore raw

RE: [PATCH 0/1] Fix for riscv vmcore issue

Quoted 8 lines

Pnina!

Quoted 3 lines · Pnina Feder

We are creating a vmcore using kexec on a Linux 6.15 RISC-V system and analyzing it with the crash tool on the host. This workflow used to work on Linux 6.14 but is now broken in 6.15.

Quoted 9 lines

Thanks for reporting this!

Quoted 4 lines

The issue is caused by a change in the kernel:
In Linux 6.15, certain memblock sections are now marked as Reserved in /proc/iomem. The kexec tool excludes all Reserved regions when generating the vmcore, so these sections are missing from the dump.

How are you collecting the /proc/vmcore file? A full set of commands would be helpful.

We’ve defined in our system that when a process crashes, we call panic().
To handle crash recovery, we're using kexec with the following command:
kexec -p /Image --initrd=/rootfs.cpio --append "console=${con} earlycon=${earlycon} no4lvl"

To simulate crash, we trigger it using:
sleep 100 & kill -6 $!

This boots into the crash kernel (kdump), where we then copy the /proc/vmcore file back to the host for analysis.

Quoted 6 lines
Quoted 3 lines

However, the kernel still uses addresses in these regions—for example, for IRQ pointers. Since the crash tool needs access to these memory areas to function correctly, their exclusion breaks the analysis.

Wdym with "IRQ pointers"? Also, what version (sha1) of crash are you using?

We are currently using crash-utility version 9.0.0 (master). From the crash analysis logs, we observed errors like:

"......
IRQ stack pointer[0] is ffffffd6fbdcc068
crash: read error: kernel virtual address: ffffffd6fbdcc068 type: "IRQ stack pointer"
.....

<read_kdump: addr: ffffffff80edf1cc paddr: 8010df1cc cnt: 4>
<readmem: ffffffd6fbdd6880, KVADDR, "runqueues entry (per_cpu)", 3456, (FOE), 55acf03963e0>

Quoted 1 line

read_kdump: addr: ffffffd6fbdd6880 paddr: 8fbdd6880 cnt: 1920<

crash: read error: kernel virtual address: ffffffd6fbdd6880 type: "runqueues entry (per_cpu)"

These failures occur consistently for addresses in the 0xffffffd000000000 region.
Upon inspection, we confirmed that the physical addresses corresponding to those virtual addresses are not present in the vmcore, as they fall under Reserved memory sections. We tested a patch to kexec-tools that prevents exclusion of the Reserved-memblock section from the vmcore. With this patch, the issue no longer occurs, and crash analysis succeeds.
Note: I suspect the same issue exists on ARM64, as both the signal.c and kexec-tools implementations are similar.

Quoted 3 lines

Thanks!
Björn

#6 lore raw

Re: [PATCH 0/1] Fix for riscv vmcore issue

Hi Pnina,

Quoted 37 lines · Pnina Feder
Quoted 7 lines

Pnina!

Quoted 3 lines · Pnina Feder

We are creating a vmcore using kexec on a Linux 6.15 RISC-V system and analyzing it with the crash tool on the host. This workflow used to work on Linux 6.14 but is now broken in 6.15.

Quoted 8 lines

Thanks for reporting this!

Quoted 4 lines

The issue is caused by a change in the kernel:
In Linux 6.15, certain memblock sections are now marked as Reserved in /proc/iomem. The kexec tool excludes all Reserved regions when generating the vmcore, so these sections are missing from the dump.

How are you collecting the /proc/vmcore file? A full set of commands would be helpful.

We’ve defined in our system that when a process crashes, we call panic().
To handle crash recovery, we're using kexec with the following command:
kexec -p /Image --initrd=/rootfs.cpio --append "console=${con} earlycon=${earlycon} no4lvl"

To simulate crash, we trigger it using:
sleep 100 & kill -6 $!

This boots into the crash kernel (kdump), where we then copy the /proc/vmcore file back to the host for analysis.

Quoted 5 lines
Quoted 3 lines

However, the kernel still uses addresses in these regions—for example, for IRQ pointers. Since the crash tool needs access to these memory areas to function correctly, their exclusion breaks the analysis.

Wdym with "IRQ pointers"? Also, what version (sha1) of crash are you using?

We are currently using crash-utility version 9.0.0 (master).
From the crash analysis logs, we observed errors like:

"......
IRQ stack pointer[0] is ffffffd6fbdcc068
crash: read error: kernel virtual address: ffffffd6fbdcc068 type: "IRQ stack pointer"
.....

Quoted 4 lines

<read_kdump: addr: ffffffff80edf1cc paddr: 8010df1cc cnt: 4>
<readmem: ffffffd6fbdd6880, KVADDR, "runqueues entry (per_cpu)", 3456, (FOE), 55acf03963e0>

Quoted 1 line

read_kdump: addr: ffffffd6fbdd6880 paddr: 8fbdd6880 cnt: 1920<

crash: read error: kernel virtual address: ffffffd6fbdd6880 type: "runqueues entry (per_cpu)"

I can't reproduce this issue on qemu, booting with sv39. I'm using the latest kexec-tools (which recently merged riscv support), crash 9.0.0 and kernel 6.16.0-rc4. Note that I'm using crash in qemu.

Are you able to reproduce this on qemu too?

Maybe that's related to the config, can you share your config?

Quoted 2 lines

These failures occur consistently for addresses in the 0xffffffd000000000 region.

FYI, this region is the direct mapping (see Documentation/arch/riscv/vm-layout.rst).

Thanks,

Alex

Quoted 6 lines

Upon inspection, we confirmed that the physical addresses corresponding to those virtual addresses are not present in the vmcore, as they fall under Reserved memory sections. We tested a patch to kexec-tools that prevents exclusion of the Reserved-memblock section from the vmcore. With this patch, the issue no longer occurs, and crash analysis succeeds.
Note: I suspect the same issue exists on ARM64, as both the signal.c and kexec-tools implementations are similar.

Quoted 2 lines

Thanks!
Björn

#7 lore raw

RE: [PATCH 0/1] Fix for riscv vmcore issue

Quoted 42 lines

Hi Pnina,

Quoted 40 lines
Quoted 15 lines

Pnina!

Quoted 3 lines · Pnina Feder

We are creating a vmcore using kexec on a Linux 6.15 RISC-V system and analyzing it with the crash tool on the host. This workflow used to work on Linux 6.14 but is now broken in 6.15.

Thanks for reporting this!

Quoted 5 lines

The issue is caused by a change in the kernel:
In Linux 6.15, certain memblock sections are now marked as Reserved
in /proc/iomem. The kexec tool excludes all Reserved regions when generating the vmcore, so these sections are missing from the dump.
How are you collecting the /proc/vmcore file? A full set of commands would be helpful.

We’ve defined in our system that when a process crashes, we call panic().
To handle crash recovery, we're using kexec with the following command:
kexec -p /Image --initrd=/rootfs.cpio --append "console=${con} earlycon=${earlycon} no4lvl"

To simulate crash, we trigger it using:
sleep 100 & kill -6 $!

This boots into the crash kernel (kdump), where we then copy the /proc/vmcore file back to the host for analysis.

Quoted 8 lines
Quoted 3 lines

However, the kernel still uses addresses in these regions—for example, for IRQ pointers. Since the crash tool needs access to these memory areas to function correctly, their exclusion breaks the analysis.

Wdym with "IRQ pointers"? Also, what version (sha1) of crash are you using?

We are currently using crash-utility version 9.0.0 (master).
From the crash analysis logs, we observed errors like:

"......
IRQ stack pointer[0] is ffffffd6fbdcc068

Quoted 1 line

crash: read error: kernel virtual address: ffffffd6fbdcc068 type: "IRQ stack pointer"

.....

Quoted 1 line

<read_kdump: addr: ffffffff80edf1cc paddr: 8010df1cc cnt: 4>

<readmem: ffffffd6fbdd6880, KVADDR, "runqueues entry (per_cpu)", 3456, (FOE), 55acf03963e0>

Quoted 7 lines
Quoted 2 lines
Quoted 1 line

read_kdump: addr: ffffffd6fbdd6880 paddr: 8fbdd6880 cnt: 1920<

crash: read error: kernel virtual address: ffffffd6fbdd6880 type: "runqueues entry (per_cpu)"

I can't reproduce this issue on qemu, booting with sv39. I'm using the latest kexec-tools (which recently merged riscv support), crash 9.0.0 and kernel 6.16.0-rc4. Note that I'm using crash in qemu.

Are you able to reproduce this on qemu too?

Yes, I am using qemu too on main and crash kernel, with latest kexec-tools, crash 9.0.0 and kernel 6.15

Quoted 1 line

Maybe that's related to the config, can you share your config?

this is my dev_config

CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
CONFIG_AUDIT=y
CONFIG_NO_HZ_IDLE=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_BPF_SYSCALL=y
CONFIG_PREEMPT_RT=y
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_PSI=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_CGROUPS=y
CONFIG_MEMCG=y
CONFIG_CGROUP_SCHED=y
CONFIG_CFS_BANDWIDTH=y
CONFIG_RT_GROUP_SCHED=y
CONFIG_CGROUP_PIDS=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_HUGETLB=y
CONFIG_CPUSETS=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_CGROUP_PERF=y
CONFIG_CGROUP_BPF=y
CONFIG_NAMESPACES=y
CONFIG_USER_NS=y
CONFIG_CHECKPOINT_RESTORE=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_EXPERT=y
CONFIG_PROFILING=y
CONFIG_KEXEC=y
CONFIG_ARCH_VIRT=y
CONFIG_NONPORTABLE=y
CONFIG_SMP=y
CONFIG_NR_CPUS=32
CONFIG_HZ_1000=y
CONFIG_CPU_IDLE=y
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_IOSCHED_BFQ=y
CONFIG_PAGE_REPORTING=y
CONFIG_PERCPU_STATS=y
CONFIG_NET=y
CONFIG_PACKET=y
CONFIG_UNIX=y
CONFIG_XFRM_USER=m
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_INET_ESP=m
CONFIG_NETWORK_SECMARK=y
CONFIG_NETFILTER=y
CONFIG_IP_NF_IPTABLES=y
CONFIG_IP_NF_FILTER=y
CONFIG_BRIDGE=m
CONFIG_BRIDGE_VLAN_FILTERING=y
CONFIG_VLAN_8021Q=m
CONFIG_NET_SCHED=y
CONFIG_NET_CLS_CGROUP=m
CONFIG_NETLINK_DIAG=y
CONFIG_NET_L3_MASTER_DEV=y
CONFIG_CGROUP_NET_PRIO=y
CONFIG_FAILOVER=y
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_MTD=y
CONFIG_MTD_BLOCK=y
CONFIG_MTD_CFI=y
CONFIG_MTD_CFI_INTELEXT=y
CONFIG_MTD_PHYSMAP=y
CONFIG_MTD_PHYSMAP_OF=y
CONFIG_BLK_DEV_LOOP=y
CONFIG_BLK_DEV_LOOP_MIN_COUNT=0
CONFIG_VIRTIO_BLK=y
CONFIG_SCSI=y
CONFIG_BLK_DEV_SD=y
CONFIG_SCSI_VIRTIO=y
CONFIG_MD=y
CONFIG_BLK_DEV_DM=y
CONFIG_NETDEVICES=y
CONFIG_MACB=y
CONFIG_PCS_XPCS=m
CONFIG_SERIO_LIBPS2=y
CONFIG_VT_HW_CONSOLE_BINDING=y
CONFIG_LEGACY_PTY_COUNT=16
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_SERIAL_OF_PLATFORM=y
CONFIG_SERIAL_EARLYCON_RISCV_SBI=y
CONFIG_VIRTIO_CONSOLE=y
CONFIG_HW_RANDOM=y
CONFIG_HW_RANDOM_VIRTIO=y
CONFIG_I2C=y
CONFIG_I2C_DESIGNWARE_CORE=y
CONFIG_SPI=y
CONFIG_PINCTRL=y
CONFIG_PINCTRL_SINGLE=y
CONFIG_GPIOLIB=y
CONFIG_GPIO_SYSFS=y
CONFIG_GPIO_DWAPB=y
CONFIG_GPIO_SIFIVE=y
CONFIG_POWER_SUPPLY=y
CONFIG_WATCHDOG=y
CONFIG_WATCHDOG_CORE=y
CONFIG_REGULATOR=y
CONFIG_REGULATOR_FIXED_VOLTAGE=y
CONFIG_BACKLIGHT_CLASS_DEVICE=m
CONFIG_SCSI_UFSHCD=y
CONFIG_SCSI_UFSHCD_PLATFORM=y
CONFIG_SCSI_UFS_DWC_TC_PLATFORM=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_DRV_M41T80=y
CONFIG_DMADEVICES=y
CONFIG_SYNC_FILE=y
CONFIG_COMMON_CLK_EYEQ=y
CONFIG_RPMSG_CHAR=y
CONFIG_RPMSG_CTRL=y
CONFIG_RPMSG_VIRTIO=y
CONFIG_RESET_CONTROLLER=y
CONFIG_RESET_SIMPLE=y
CONFIG_GENERIC_PHY=y
CONFIG_EXT4_FS=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y
CONFIG_MSDOS_FS=y
CONFIG_VFAT_FS=y
CONFIG_TMPFS=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_HUGETLBFS=y
CONFIG_KEYS=y
CONFIG_SECURITY=y
CONFIG_SECURITYFS=y
CONFIG_SECURITY_NETWORK=y
CONFIG_SECURITY_PATH=y
CONFIG_CRYPTO_RSA=y
CONFIG_CRYPTO_ECB=y
CONFIG_CRYPTO_BLAKE2B=m
CONFIG_CRYPTO_XXHASH=m
CONFIG_CRYPTO_USER_API_HASH=y
CONFIG_CRC_CCITT=m
CONFIG_CRC_ITU_T=y
CONFIG_CRC7=y
CONFIG_LIBCRC32C=m
CONFIG_PRINTK_TIME=y
CONFIG_DYNAMIC_DEBUG=y
CONFIG_DEBUG_INFO_DWARF5=y
CONFIG_DEBUG_FS=y
CONFIG_DEBUG_PAGEALLOC=y
CONFIG_PTDUMP_DEBUGFS=y
CONFIG_SCHED_STACK_END_CHECK=y
CONFIG_DEBUG_VM=y
CONFIG_DEBUG_VM_PGFLAGS=y
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_DEBUG_PER_CPU_MAPS=y
CONFIG_SOFTLOCKUP_DETECTOR=y
CONFIG_WQ_WATCHDOG=y
CONFIG_DEBUG_RT_MUTEXES=y
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_ATOMIC_SLEEP=y
CONFIG_DEBUG_LIST=y
CONFIG_DEBUG_PLIST=y
CONFIG_DEBUG_SG=y
CONFIG_RCU_EQS_DEBUG=y
CONFIG_MEMTEST=y

Quoted 16 lines
Quoted 1 line

These failures occur consistently for addresses in the 0xffffffd000000000 region.

FYI, this region is the direct mapping (see Documentation/arch/riscv/vm-layout.rst).

Thanks,

Alex

Quoted 6 lines

Upon inspection, we confirmed that the physical addresses corresponding to those virtual addresses are not present in the vmcore, as they fall under Reserved memory sections. We tested a patch to kexec-tools that prevents exclusion of the Reserved-memblock section from the vmcore. With this patch, the issue no longer occurs, and crash analysis succeeds.
Note: I suspect the same issue exists on ARM64, as both the signal.c and kexec-tools implementations are similar.

Quoted 2 lines

Thanks!
Björn

#8 lore raw

RE: [PATCH 0/1] Fix for riscv vmcore issue

Quoted 214 lines
Quoted 42 lines
Quoted 42 lines

Hi Pnina,

Quoted 15 lines

Pnina!

Quoted 3 lines · Pnina Feder

We are creating a vmcore using kexec on a Linux 6.15 RISC-V system and analyzing it with the crash tool on the host. This workflow used to work on Linux 6.14 but is now broken in 6.15.

Thanks for reporting this!

Quoted 5 lines

The issue is caused by a change in the kernel:
In Linux 6.15, certain memblock sections are now marked as Reserved
in /proc/iomem. The kexec tool excludes all Reserved regions when generating the vmcore, so these sections are missing from the dump.
How are you collecting the /proc/vmcore file? A full set of commands would be helpful.

We’ve defined in our system that when a process crashes, we call panic().
To handle crash recovery, we're using kexec with the following command:
kexec -p /Image --initrd=/rootfs.cpio --append "console=${con} earlycon=${earlycon} no4lvl"

To simulate crash, we trigger it using:
sleep 100 & kill -6 $!

This boots into the crash kernel (kdump), where we then copy the /proc/vmcore file back to the host for analysis.

Quoted 8 lines
Quoted 3 lines

However, the kernel still uses addresses in these regions—for example, for IRQ pointers. Since the crash tool needs access to these memory areas to function correctly, their exclusion breaks the analysis.

Wdym with "IRQ pointers"? Also, what version (sha1) of crash are you using?

We are currently using crash-utility version 9.0.0 (master).
From the crash analysis logs, we observed errors like:

"......
IRQ stack pointer[0] is ffffffd6fbdcc068

Quoted 1 line

crash: read error: kernel virtual address: ffffffd6fbdcc068 type: "IRQ stack pointer"

.....

Quoted 1 line

<read_kdump: addr: ffffffff80edf1cc paddr: 8010df1cc cnt: 4>

<readmem: ffffffd6fbdd6880, KVADDR, "runqueues entry (per_cpu)", 3456, (FOE), 55acf03963e0>

Quoted 7 lines
Quoted 2 lines
Quoted 1 line

read_kdump: addr: ffffffd6fbdd6880 paddr: 8fbdd6880 cnt: 1920<

crash: read error: kernel virtual address: ffffffd6fbdd6880 type: "runqueues entry (per_cpu)"

I can't reproduce this issue on qemu, booting with sv39. I'm using the latest kexec-tools (which recently merged riscv .support), crash 9.0.0 and kernel 6.16.0-rc4. Note that I'm using crash in qemu.

Are you able to reproduce this on qemu too?

Yes, I am using qemu too on main and crash kernel, with latest kexec-tools, crash 9.0.0 and kernel 6.15

Quoted 1 line

Maybe that's related to the config, can you share your config?

this is my dev_config

CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
CONFIG_AUDIT=y
CONFIG_NO_HZ_IDLE=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_BPF_SYSCALL=y
CONFIG_PREEMPT_RT=y
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_PSI=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_CGROUPS=y
CONFIG_MEMCG=y
CONFIG_CGROUP_SCHED=y
CONFIG_CFS_BANDWIDTH=y
CONFIG_RT_GROUP_SCHED=y
CONFIG_CGROUP_PIDS=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_HUGETLB=y
CONFIG_CPUSETS=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_CGROUP_PERF=y
CONFIG_CGROUP_BPF=y
CONFIG_NAMESPACES=y
CONFIG_USER_NS=y
CONFIG_CHECKPOINT_RESTORE=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_EXPERT=y
CONFIG_PROFILING=y
CONFIG_KEXEC=y
CONFIG_ARCH_VIRT=y
CONFIG_NONPORTABLE=y
CONFIG_SMP=y
CONFIG_NR_CPUS=32
CONFIG_HZ_1000=y
CONFIG_CPU_IDLE=y
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_IOSCHED_BFQ=y
CONFIG_PAGE_REPORTING=y
CONFIG_PERCPU_STATS=y
CONFIG_NET=y
CONFIG_PACKET=y
CONFIG_UNIX=y
CONFIG_XFRM_USER=m
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_INET_ESP=m
CONFIG_NETWORK_SECMARK=y
CONFIG_NETFILTER=y
CONFIG_IP_NF_IPTABLES=y
CONFIG_IP_NF_FILTER=y
CONFIG_BRIDGE=m
CONFIG_BRIDGE_VLAN_FILTERING=y
CONFIG_VLAN_8021Q=m
CONFIG_NET_SCHED=y
CONFIG_NET_CLS_CGROUP=m
CONFIG_NETLINK_DIAG=y
CONFIG_NET_L3_MASTER_DEV=y
CONFIG_CGROUP_NET_PRIO=y
CONFIG_FAILOVER=y
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_MTD=y
CONFIG_MTD_BLOCK=y
CONFIG_MTD_CFI=y
CONFIG_MTD_CFI_INTELEXT=y
CONFIG_MTD_PHYSMAP=y
CONFIG_MTD_PHYSMAP_OF=y
CONFIG_BLK_DEV_LOOP=y
CONFIG_BLK_DEV_LOOP_MIN_COUNT=0
CONFIG_VIRTIO_BLK=y
CONFIG_SCSI=y
CONFIG_BLK_DEV_SD=y
CONFIG_SCSI_VIRTIO=y
CONFIG_MD=y
CONFIG_BLK_DEV_DM=y
CONFIG_NETDEVICES=y
CONFIG_MACB=y
CONFIG_PCS_XPCS=m
CONFIG_SERIO_LIBPS2=y
CONFIG_VT_HW_CONSOLE_BINDING=y
CONFIG_LEGACY_PTY_COUNT=16
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_SERIAL_OF_PLATFORM=y
CONFIG_SERIAL_EARLYCON_RISCV_SBI=y
CONFIG_VIRTIO_CONSOLE=y
CONFIG_HW_RANDOM=y
CONFIG_HW_RANDOM_VIRTIO=y
CONFIG_I2C=y
CONFIG_I2C_DESIGNWARE_CORE=y
CONFIG_SPI=y
CONFIG_PINCTRL=y
CONFIG_PINCTRL_SINGLE=y
CONFIG_GPIOLIB=y
CONFIG_GPIO_SYSFS=y
CONFIG_GPIO_DWAPB=y
CONFIG_GPIO_SIFIVE=y
CONFIG_POWER_SUPPLY=y
CONFIG_WATCHDOG=y
CONFIG_WATCHDOG_CORE=y
CONFIG_REGULATOR=y
CONFIG_REGULATOR_FIXED_VOLTAGE=y
CONFIG_BACKLIGHT_CLASS_DEVICE=m
CONFIG_SCSI_UFSHCD=y
CONFIG_SCSI_UFSHCD_PLATFORM=y
CONFIG_SCSI_UFS_DWC_TC_PLATFORM=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_DRV_M41T80=y
CONFIG_DMADEVICES=y
CONFIG_SYNC_FILE=y
CONFIG_COMMON_CLK_EYEQ=y
CONFIG_RPMSG_CHAR=y
CONFIG_RPMSG_CTRL=y
CONFIG_RPMSG_VIRTIO=y
CONFIG_RESET_CONTROLLER=y
CONFIG_RESET_SIMPLE=y
CONFIG_GENERIC_PHY=y
CONFIG_EXT4_FS=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y
CONFIG_MSDOS_FS=y
CONFIG_VFAT_FS=y
CONFIG_TMPFS=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_HUGETLBFS=y
CONFIG_KEYS=y
CONFIG_SECURITY=y
CONFIG_SECURITYFS=y
CONFIG_SECURITY_NETWORK=y
CONFIG_SECURITY_PATH=y
CONFIG_CRYPTO_RSA=y
CONFIG_CRYPTO_ECB=y
CONFIG_CRYPTO_BLAKE2B=m
CONFIG_CRYPTO_XXHASH=m
CONFIG_CRYPTO_USER_API_HASH=y
CONFIG_CRC_CCITT=m
CONFIG_CRC_ITU_T=y
CONFIG_CRC7=y
CONFIG_LIBCRC32C=m
CONFIG_PRINTK_TIME=y
CONFIG_DYNAMIC_DEBUG=y
CONFIG_DEBUG_INFO_DWARF5=y
CONFIG_DEBUG_FS=y
CONFIG_DEBUG_PAGEALLOC=y
CONFIG_PTDUMP_DEBUGFS=y
CONFIG_SCHED_STACK_END_CHECK=y
CONFIG_DEBUG_VM=y
CONFIG_DEBUG_VM_PGFLAGS=y
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_DEBUG_PER_CPU_MAPS=y
CONFIG_SOFTLOCKUP_DETECTOR=y
CONFIG_WQ_WATCHDOG=y

Quoted 18 lines

CONFIG_DEBUG_RT_MUTEXES=y
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_ATOMIC_SLEEP=y
CONFIG_DEBUG_LIST=y
CONFIG_DEBUG_PLIST=y
CONFIG_DEBUG_SG=y
CONFIG_RCU_EQS_DEBUG=y
CONFIG_MEMTEST=y

Quoted 9 lines
Quoted 1 line

These failures occur consistently for addresses in the 0xffffffd000000000 region.

FYI, this region is the direct mapping (see Documentation/arch/riscv/vm-layout.rst).

Thanks,

Alex

Hi Alex!

Do I have something to try or help to process this issue? maybe, can you give your Config and I will try it on my system? Any more information I can share?

Thanks a lot,
Pnina

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Upon inspection, we confirmed that the physical addresses corresponding to those virtual addresses are not present in the vmcore, as they fall under Reserved memory sections. We tested a patch to kexec-tools that prevents exclusion of the Reserved-memblock section from the vmcore. With this patch, the issue no longer occurs, and crash analysis succeeds.
Note: I suspect the same issue exists on ARM64, as both the signal.c and kexec-tools implementations are similar.

Quoted 2 lines

Thanks!
Björn

#9 lore raw

Re: [PATCH 0/1] Fix for riscv vmcore issue

Hi Pnina,

Quoted 224 lines · Pnina Feder
Quoted 224 lines
Quoted 38 lines
Quoted 38 lines

Hi Pnina,

Quoted 14 lines

Pnina!

Quoted 3 lines · Pnina Feder

We are creating a vmcore using kexec on a Linux 6.15 RISC-V system and analyzing it with the crash tool on the host. This workflow used to work on Linux 6.14 but is now broken in 6.15.

Thanks for reporting this!

Quoted 5 lines

The issue is caused by a change in the kernel:
In Linux 6.15, certain memblock sections are now marked as Reserved
in /proc/iomem. The kexec tool excludes all Reserved regions when generating the vmcore, so these sections are missing from the dump.
How are you collecting the /proc/vmcore file? A full set of commands would be helpful.

We’ve defined in our system that when a process crashes, we call panic().
To handle crash recovery, we're using kexec with the following command:
kexec -p /Image --initrd=/rootfs.cpio --append "console=${con} earlycon=${earlycon} no4lvl"

To simulate crash, we trigger it using:
sleep 100 & kill -6 $!

This boots into the crash kernel (kdump), where we then copy the /proc/vmcore file back to the host for analysis.

Quoted 8 lines
Quoted 3 lines

However, the kernel still uses addresses in these regions—for example, for IRQ pointers. Since the crash tool needs access to these memory areas to function correctly, their exclusion breaks the analysis.

Wdym with "IRQ pointers"? Also, what version (sha1) of crash are you using?

We are currently using crash-utility version 9.0.0 (master).
From the crash analysis logs, we observed errors like:

"......
IRQ stack pointer[0] is ffffffd6fbdcc068

Quoted 1 line

crash: read error: kernel virtual address: ffffffd6fbdcc068 type: "IRQ stack pointer"

.....

Quoted 1 line

<read_kdump: addr: ffffffff80edf1cc paddr: 8010df1cc cnt: 4>

Quoted 8 lines
Quoted 4 lines

<readmem: ffffffd6fbdd6880, KVADDR, "runqueues entry (per_cpu)", 3456, (FOE), 55acf03963e0>

Quoted 1 line

read_kdump: addr: ffffffd6fbdd6880 paddr: 8fbdd6880 cnt: 1920<

crash: read error: kernel virtual address: ffffffd6fbdd6880 type: "runqueues entry (per_cpu)"

I can't reproduce this issue on qemu, booting with sv39. I'm using the latest kexec-tools (which recently merged riscv .support), crash 9.0.0 and kernel 6.16.0-rc4. Note that I'm using crash in qemu.

Are you able to reproduce this on qemu too?

Yes, I am using qemu too on main and crash kernel, with latest kexec-tools, crash 9.0.0 and kernel 6.15

Quoted 1 line

Maybe that's related to the config, can you share your config?

this is my dev_config

CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
CONFIG_AUDIT=y
CONFIG_NO_HZ_IDLE=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_BPF_SYSCALL=y
CONFIG_PREEMPT_RT=y
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_PSI=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_CGROUPS=y
CONFIG_MEMCG=y
CONFIG_CGROUP_SCHED=y
CONFIG_CFS_BANDWIDTH=y
CONFIG_RT_GROUP_SCHED=y
CONFIG_CGROUP_PIDS=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_HUGETLB=y
CONFIG_CPUSETS=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_CGROUP_PERF=y
CONFIG_CGROUP_BPF=y
CONFIG_NAMESPACES=y
CONFIG_USER_NS=y
CONFIG_CHECKPOINT_RESTORE=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_EXPERT=y
CONFIG_PROFILING=y
CONFIG_KEXEC=y
CONFIG_ARCH_VIRT=y
CONFIG_NONPORTABLE=y
CONFIG_SMP=y
CONFIG_NR_CPUS=32
CONFIG_HZ_1000=y
CONFIG_CPU_IDLE=y
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_IOSCHED_BFQ=y
CONFIG_PAGE_REPORTING=y
CONFIG_PERCPU_STATS=y
CONFIG_NET=y
CONFIG_PACKET=y
CONFIG_UNIX=y
CONFIG_XFRM_USER=m
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_INET_ESP=m
CONFIG_NETWORK_SECMARK=y
CONFIG_NETFILTER=y
CONFIG_IP_NF_IPTABLES=y
CONFIG_IP_NF_FILTER=y
CONFIG_BRIDGE=m
CONFIG_BRIDGE_VLAN_FILTERING=y
CONFIG_VLAN_8021Q=m
CONFIG_NET_SCHED=y
CONFIG_NET_CLS_CGROUP=m
CONFIG_NETLINK_DIAG=y
CONFIG_NET_L3_MASTER_DEV=y
CONFIG_CGROUP_NET_PRIO=y
CONFIG_FAILOVER=y
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_MTD=y
CONFIG_MTD_BLOCK=y
CONFIG_MTD_CFI=y
CONFIG_MTD_CFI_INTELEXT=y
CONFIG_MTD_PHYSMAP=y
CONFIG_MTD_PHYSMAP_OF=y
CONFIG_BLK_DEV_LOOP=y
CONFIG_BLK_DEV_LOOP_MIN_COUNT=0
CONFIG_VIRTIO_BLK=y
CONFIG_SCSI=y
CONFIG_BLK_DEV_SD=y
CONFIG_SCSI_VIRTIO=y
CONFIG_MD=y
CONFIG_BLK_DEV_DM=y
CONFIG_NETDEVICES=y
CONFIG_MACB=y
CONFIG_PCS_XPCS=m
CONFIG_SERIO_LIBPS2=y
CONFIG_VT_HW_CONSOLE_BINDING=y
CONFIG_LEGACY_PTY_COUNT=16
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_SERIAL_OF_PLATFORM=y
CONFIG_SERIAL_EARLYCON_RISCV_SBI=y
CONFIG_VIRTIO_CONSOLE=y
CONFIG_HW_RANDOM=y
CONFIG_HW_RANDOM_VIRTIO=y
CONFIG_I2C=y
CONFIG_I2C_DESIGNWARE_CORE=y
CONFIG_SPI=y
CONFIG_PINCTRL=y
CONFIG_PINCTRL_SINGLE=y
CONFIG_GPIOLIB=y
CONFIG_GPIO_SYSFS=y
CONFIG_GPIO_DWAPB=y
CONFIG_GPIO_SIFIVE=y
CONFIG_POWER_SUPPLY=y
CONFIG_WATCHDOG=y
CONFIG_WATCHDOG_CORE=y
CONFIG_REGULATOR=y
CONFIG_REGULATOR_FIXED_VOLTAGE=y
CONFIG_BACKLIGHT_CLASS_DEVICE=m
CONFIG_SCSI_UFSHCD=y
CONFIG_SCSI_UFSHCD_PLATFORM=y
CONFIG_SCSI_UFS_DWC_TC_PLATFORM=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_DRV_M41T80=y
CONFIG_DMADEVICES=y
CONFIG_SYNC_FILE=y
CONFIG_COMMON_CLK_EYEQ=y
CONFIG_RPMSG_CHAR=y
CONFIG_RPMSG_CTRL=y
CONFIG_RPMSG_VIRTIO=y
CONFIG_RESET_CONTROLLER=y
CONFIG_RESET_SIMPLE=y
CONFIG_GENERIC_PHY=y
CONFIG_EXT4_FS=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y
CONFIG_MSDOS_FS=y
CONFIG_VFAT_FS=y
CONFIG_TMPFS=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_HUGETLBFS=y
CONFIG_KEYS=y
CONFIG_SECURITY=y
CONFIG_SECURITYFS=y
CONFIG_SECURITY_NETWORK=y
CONFIG_SECURITY_PATH=y
CONFIG_CRYPTO_RSA=y
CONFIG_CRYPTO_ECB=y
CONFIG_CRYPTO_BLAKE2B=m
CONFIG_CRYPTO_XXHASH=m
CONFIG_CRYPTO_USER_API_HASH=y
CONFIG_CRC_CCITT=m
CONFIG_CRC_ITU_T=y
CONFIG_CRC7=y
CONFIG_LIBCRC32C=m
CONFIG_PRINTK_TIME=y
CONFIG_DYNAMIC_DEBUG=y
CONFIG_DEBUG_INFO_DWARF5=y
CONFIG_DEBUG_FS=y
CONFIG_DEBUG_PAGEALLOC=y
CONFIG_PTDUMP_DEBUGFS=y
CONFIG_SCHED_STACK_END_CHECK=y
CONFIG_DEBUG_VM=y
CONFIG_DEBUG_VM_PGFLAGS=y
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_DEBUG_PER_CPU_MAPS=y
CONFIG_SOFTLOCKUP_DETECTOR=y
CONFIG_WQ_WATCHDOG=y
CONFIG_DEBUG_RT_MUTEXES=y
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_ATOMIC_SLEEP=y
CONFIG_DEBUG_LIST=y
CONFIG_DEBUG_PLIST=y
CONFIG_DEBUG_SG=y
CONFIG_RCU_EQS_DEBUG=y
CONFIG_MEMTEST=y

Quoted 6 lines
Quoted 1 line

These failures occur consistently for addresses in the 0xffffffd000000000 region.

FYI, this region is the direct mapping (see Documentation/arch/riscv/vm-layout.rst).

Thanks,

Quoted 7 lines
Quoted 2 lines
Quoted 2 lines

Alex

Hi Alex!

Do I have something to try or help to process this issue? maybe, can you give your Config and I will try it on my system? Any more information I can share?

So I'm able to reproduce your issue with your config, it only happens with kexec_load(), not kexec_file_load().

Your patch does not fix the problem for me, makedumpfile still fails. I spent quite some time looking for the code that parses the memory regions and exposes them as PT_LOAD segments in vmcore, but I did not find it, do you know where that happens for kexec_load()?

Thanks,

Alex

Quoted 14 lines

Thanks a lot,
Pnina

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Upon inspection, we confirmed that the physical addresses corresponding to those virtual addresses are not present in the vmcore, as they fall under Reserved memory sections. We tested a patch to kexec-tools that prevents exclusion of the Reserved-memblock section from the vmcore. With this patch, the issue no longer occurs, and crash analysis succeeds.
Note: I suspect the same issue exists on ARM64, as both the signal.c and kexec-tools implementations are similar.

Quoted 2 lines

Thanks!
Björn

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#10 lore raw

RE: [PATCH 0/1] Fix for riscv vmcore issue

Hi Alex,
Thank you for your response!

Quoted 237 lines

Hi Pnina,

Quoted 225 lines · Pnina Feder
Quoted 225 lines
Quoted 38 lines
Quoted 38 lines

Hi Pnina,

Quoted 14 lines

Pnina!

Quoted 3 lines · Pnina Feder

We are creating a vmcore using kexec on a Linux 6.15 RISC-V system and analyzing it with the crash tool on the host. This workflow used to work on Linux 6.14 but is now broken in 6.15.

Thanks for reporting this!

Quoted 5 lines

The issue is caused by a change in the kernel:
In Linux 6.15, certain memblock sections are now marked as
Reserved in /proc/iomem. The kexec tool excludes all Reserved
regions when generating the vmcore, so these sections are missing from the dump. How are you collecting the /proc/vmcore file? A full set of commands would be helpful.

We’ve defined in our system that when a process crashes, we call panic().
To handle crash recovery, we're using kexec with the following command:
kexec -p /Image --initrd=/rootfs.cpio --append "console=${con} earlycon=${earlycon} no4lvl"

To simulate crash, we trigger it using:
sleep 100 & kill -6 $!

This boots into the crash kernel (kdump), where we then copy the /proc/vmcore file back to the host for analysis.

Quoted 8 lines
Quoted 3 lines

However, the kernel still uses addresses in these regions—for example, for IRQ pointers. Since the crash tool needs access to these memory areas to function correctly, their exclusion breaks the analysis.

Wdym with "IRQ pointers"? Also, what version (sha1) of crash are you using?

We are currently using crash-utility version 9.0.0 (master).
From the crash analysis logs, we observed errors like:

"......
IRQ stack pointer[0] is ffffffd6fbdcc068

Quoted 1 line

crash: read error: kernel virtual address: ffffffd6fbdcc068 type: "IRQ stack pointer"

.....

Quoted 1 line

<read_kdump: addr: ffffffff80edf1cc paddr: 8010df1cc cnt: 4>

Quoted 8 lines
Quoted 4 lines

<readmem: ffffffd6fbdd6880, KVADDR, "runqueues entry (per_cpu)", 3456, (FOE), 55acf03963e0>

Quoted 1 line

read_kdump: addr: ffffffd6fbdd6880 paddr: 8fbdd6880 cnt: 1920<

crash: read error: kernel virtual address: ffffffd6fbdd6880 type: "runqueues entry (per_cpu)"

I can't reproduce this issue on qemu, booting with sv39. I'm using the latest kexec-tools (which recently merged riscv .support), crash 9.0.0 and kernel 6.16.0-rc4. Note that I'm using crash in qemu.

Are you able to reproduce this on qemu too?

Yes, I am using qemu too on main and crash kernel, with latest kexec-tools, crash 9.0.0 and kernel 6.15

Quoted 1 line

Maybe that's related to the config, can you share your config?

this is my dev_config

CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
CONFIG_AUDIT=y
CONFIG_NO_HZ_IDLE=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_BPF_SYSCALL=y
CONFIG_PREEMPT_RT=y
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_PSI=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_CGROUPS=y
CONFIG_MEMCG=y
CONFIG_CGROUP_SCHED=y
CONFIG_CFS_BANDWIDTH=y
CONFIG_RT_GROUP_SCHED=y
CONFIG_CGROUP_PIDS=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_HUGETLB=y
CONFIG_CPUSETS=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_CGROUP_PERF=y
CONFIG_CGROUP_BPF=y
CONFIG_NAMESPACES=y
CONFIG_USER_NS=y
CONFIG_CHECKPOINT_RESTORE=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_EXPERT=y
CONFIG_PROFILING=y
CONFIG_KEXEC=y
CONFIG_ARCH_VIRT=y
CONFIG_NONPORTABLE=y
CONFIG_SMP=y
CONFIG_NR_CPUS=32
CONFIG_HZ_1000=y
CONFIG_CPU_IDLE=y
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_IOSCHED_BFQ=y
CONFIG_PAGE_REPORTING=y
CONFIG_PERCPU_STATS=y
CONFIG_NET=y
CONFIG_PACKET=y
CONFIG_UNIX=y
CONFIG_XFRM_USER=m
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_INET_ESP=m
CONFIG_NETWORK_SECMARK=y
CONFIG_NETFILTER=y
CONFIG_IP_NF_IPTABLES=y
CONFIG_IP_NF_FILTER=y
CONFIG_BRIDGE=m
CONFIG_BRIDGE_VLAN_FILTERING=y
CONFIG_VLAN_8021Q=m
CONFIG_NET_SCHED=y
CONFIG_NET_CLS_CGROUP=m
CONFIG_NETLINK_DIAG=y
CONFIG_NET_L3_MASTER_DEV=y
CONFIG_CGROUP_NET_PRIO=y
CONFIG_FAILOVER=y
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_MTD=y
CONFIG_MTD_BLOCK=y
CONFIG_MTD_CFI=y
CONFIG_MTD_CFI_INTELEXT=y
CONFIG_MTD_PHYSMAP=y
CONFIG_MTD_PHYSMAP_OF=y
CONFIG_BLK_DEV_LOOP=y
CONFIG_BLK_DEV_LOOP_MIN_COUNT=0
CONFIG_VIRTIO_BLK=y
CONFIG_SCSI=y
CONFIG_BLK_DEV_SD=y
CONFIG_SCSI_VIRTIO=y
CONFIG_MD=y
CONFIG_BLK_DEV_DM=y
CONFIG_NETDEVICES=y
CONFIG_MACB=y
CONFIG_PCS_XPCS=m
CONFIG_SERIO_LIBPS2=y
CONFIG_VT_HW_CONSOLE_BINDING=y
CONFIG_LEGACY_PTY_COUNT=16
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_SERIAL_OF_PLATFORM=y
CONFIG_SERIAL_EARLYCON_RISCV_SBI=y
CONFIG_VIRTIO_CONSOLE=y
CONFIG_HW_RANDOM=y
CONFIG_HW_RANDOM_VIRTIO=y
CONFIG_I2C=y
CONFIG_I2C_DESIGNWARE_CORE=y
CONFIG_SPI=y
CONFIG_PINCTRL=y
CONFIG_PINCTRL_SINGLE=y
CONFIG_GPIOLIB=y
CONFIG_GPIO_SYSFS=y
CONFIG_GPIO_DWAPB=y
CONFIG_GPIO_SIFIVE=y
CONFIG_POWER_SUPPLY=y
CONFIG_WATCHDOG=y
CONFIG_WATCHDOG_CORE=y
CONFIG_REGULATOR=y
CONFIG_REGULATOR_FIXED_VOLTAGE=y
CONFIG_BACKLIGHT_CLASS_DEVICE=m
CONFIG_SCSI_UFSHCD=y
CONFIG_SCSI_UFSHCD_PLATFORM=y
CONFIG_SCSI_UFS_DWC_TC_PLATFORM=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_DRV_M41T80=y
CONFIG_DMADEVICES=y
CONFIG_SYNC_FILE=y
CONFIG_COMMON_CLK_EYEQ=y
CONFIG_RPMSG_CHAR=y
CONFIG_RPMSG_CTRL=y
CONFIG_RPMSG_VIRTIO=y
CONFIG_RESET_CONTROLLER=y
CONFIG_RESET_SIMPLE=y
CONFIG_GENERIC_PHY=y
CONFIG_EXT4_FS=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y
CONFIG_MSDOS_FS=y
CONFIG_VFAT_FS=y
CONFIG_TMPFS=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_HUGETLBFS=y
CONFIG_KEYS=y
CONFIG_SECURITY=y
CONFIG_SECURITYFS=y
CONFIG_SECURITY_NETWORK=y
CONFIG_SECURITY_PATH=y
CONFIG_CRYPTO_RSA=y
CONFIG_CRYPTO_ECB=y
CONFIG_CRYPTO_BLAKE2B=m
CONFIG_CRYPTO_XXHASH=m
CONFIG_CRYPTO_USER_API_HASH=y
CONFIG_CRC_CCITT=m
CONFIG_CRC_ITU_T=y
CONFIG_CRC7=y
CONFIG_LIBCRC32C=m
CONFIG_PRINTK_TIME=y
CONFIG_DYNAMIC_DEBUG=y
CONFIG_DEBUG_INFO_DWARF5=y
CONFIG_DEBUG_FS=y
CONFIG_DEBUG_PAGEALLOC=y
CONFIG_PTDUMP_DEBUGFS=y
CONFIG_SCHED_STACK_END_CHECK=y
CONFIG_DEBUG_VM=y
CONFIG_DEBUG_VM_PGFLAGS=y
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_DEBUG_PER_CPU_MAPS=y
CONFIG_SOFTLOCKUP_DETECTOR=y
CONFIG_WQ_WATCHDOG=y
CONFIG_DEBUG_RT_MUTEXES=y
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_ATOMIC_SLEEP=y
CONFIG_DEBUG_LIST=y
CONFIG_DEBUG_PLIST=y
CONFIG_DEBUG_SG=y
CONFIG_RCU_EQS_DEBUG=y
CONFIG_MEMTEST=y

Quoted 6 lines
Quoted 1 line

These failures occur consistently for addresses in the 0xffffffd000000000 region.

FYI, this region is the direct mapping (see Documentation/arch/riscv/vm-layout.rst).

Thanks,

Quoted 7 lines
Quoted 2 lines
Quoted 2 lines

Alex

Hi Alex!

Do I have something to try or help to process this issue? maybe, can you give your Config and I will try it on my system? Any more information I can share?

Quoted 9 lines

So I'm able to reproduce your issue with your config, it only happens with kexec_load(), not kexec_file_load().

Your patch does not fix the problem for me, makedumpfile still fails. I spent quite some time looking for the code that parses the memory regions and exposes them as PT_LOAD segments in vmcore, but I did not find it, do you know where that happens for kexec_load()?

Thanks,

Alex

The code that parses memory regions is located in kexec-tools.

To fix the issue with missing memory regions in the vmcore, we need to ensure that kexec-tools does not exclude the Reserved-memblock sections when generating ELF headers for kexec_load().

I’ve added a patch to handle this, and plan to submit it upstream to kexec-tools once this approach is confirmed and approved.

Kexec-tools patch:
---

kexec/arch/riscv/kexec-riscv.c +5 -2
index f34b468..1a93b51 100644
@@ -421,8 +421,11 @@ static bool to_be_excluded(char *str, unsigned long long start, unsigned long lo
 	    !strncmp(str, KERNEL_CODE, strlen(KERNEL_CODE)) ||
 	    !strncmp(str, KERNEL_DATA, strlen(KERNEL_DATA)))
 		return false;
-	else
-		return true;
+
+	if (!strncmp(str, "Reserved-memblock", strlen("Reserved-memblock")))
+		return false;
+
+	return true;
 }
 
 int get_memory_ranges(struct memory_range **range, int *num_ranges,

---

With this patch, the kexec-tools will no longer exclude the Reserved-memblock regions, allowing the crash tool to access the necessary memory areas for analysis.

Thanks,
Pnina

Quoted 14 lines
Quoted 14 lines

Thanks a lot,
Pnina

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Upon inspection, we confirmed that the physical addresses corresponding to those virtual addresses are not present in the vmcore, as they fall under Reserved memory sections. We tested a patch to kexec-tools that prevents exclusion of the Reserved-memblock section from the vmcore. With this patch, the issue no longer occurs, and crash analysis succeeds.
Note: I suspect the same issue exists on ARM64, as both the signal.c and kexec-tools implementations are similar.

Quoted 2 lines

Thanks!
Björn

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#11 lore raw

Re: [PATCH 0/1] Fix for riscv vmcore issue

+cc Simon

Quoted 239 lines · Pnina Feder

Hi Alex,
Thank you for your response!

Quoted 236 lines

Hi Pnina,

Quoted 223 lines · Pnina Feder
Quoted 223 lines
Quoted 37 lines
Quoted 37 lines

Hi Pnina,

Quoted 14 lines

Pnina!

Quoted 3 lines · Pnina Feder

We are creating a vmcore using kexec on a Linux 6.15 RISC-V system and analyzing it with the crash tool on the host. This workflow used to work on Linux 6.14 but is now broken in 6.15.

Thanks for reporting this!

Quoted 5 lines

The issue is caused by a change in the kernel:
In Linux 6.15, certain memblock sections are now marked as
Reserved in /proc/iomem. The kexec tool excludes all Reserved
regions when generating the vmcore, so these sections are missing from the dump. How are you collecting the /proc/vmcore file? A full set of commands would be helpful.

We’ve defined in our system that when a process crashes, we call panic().
To handle crash recovery, we're using kexec with the following command:
kexec -p /Image --initrd=/rootfs.cpio --append "console=${con} earlycon=${earlycon} no4lvl"

To simulate crash, we trigger it using:
sleep 100 & kill -6 $!

This boots into the crash kernel (kdump), where we then copy the /proc/vmcore file back to the host for analysis.

Quoted 8 lines
Quoted 3 lines

However, the kernel still uses addresses in these regions—for example, for IRQ pointers. Since the crash tool needs access to these memory areas to function correctly, their exclusion breaks the analysis.

Wdym with "IRQ pointers"? Also, what version (sha1) of crash are you using?

We are currently using crash-utility version 9.0.0 (master).
From the crash analysis logs, we observed errors like:

"......
IRQ stack pointer[0] is ffffffd6fbdcc068

Quoted 1 line

crash: read error: kernel virtual address: ffffffd6fbdcc068 type: "IRQ stack pointer"

.....

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

<read_kdump: addr: ffffffff80edf1cc paddr: 8010df1cc cnt: 4>

<readmem: ffffffd6fbdd6880, KVADDR, "runqueues entry (per_cpu)", 3456, (FOE), 55acf03963e0>

Quoted 1 line

read_kdump: addr: ffffffd6fbdd6880 paddr: 8fbdd6880 cnt: 1920<

crash: read error: kernel virtual address: ffffffd6fbdd6880 type: "runqueues entry (per_cpu)"

I can't reproduce this issue on qemu, booting with sv39. I'm using the latest kexec-tools (which recently merged riscv .support), crash 9.0.0 and kernel 6.16.0-rc4. Note that I'm using crash in qemu.

Are you able to reproduce this on qemu too?

Yes, I am using qemu too on main and crash kernel, with latest kexec-tools, crash 9.0.0 and kernel 6.15

Quoted 1 line

Maybe that's related to the config, can you share your config?

this is my dev_config

CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
CONFIG_AUDIT=y
CONFIG_NO_HZ_IDLE=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_BPF_SYSCALL=y
CONFIG_PREEMPT_RT=y
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_PSI=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_CGROUPS=y
CONFIG_MEMCG=y
CONFIG_CGROUP_SCHED=y
CONFIG_CFS_BANDWIDTH=y
CONFIG_RT_GROUP_SCHED=y
CONFIG_CGROUP_PIDS=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_HUGETLB=y
CONFIG_CPUSETS=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_CGROUP_PERF=y
CONFIG_CGROUP_BPF=y
CONFIG_NAMESPACES=y
CONFIG_USER_NS=y
CONFIG_CHECKPOINT_RESTORE=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_EXPERT=y
CONFIG_PROFILING=y
CONFIG_KEXEC=y
CONFIG_ARCH_VIRT=y
CONFIG_NONPORTABLE=y
CONFIG_SMP=y
CONFIG_NR_CPUS=32
CONFIG_HZ_1000=y
CONFIG_CPU_IDLE=y
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_IOSCHED_BFQ=y
CONFIG_PAGE_REPORTING=y
CONFIG_PERCPU_STATS=y
CONFIG_NET=y
CONFIG_PACKET=y
CONFIG_UNIX=y
CONFIG_XFRM_USER=m
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_INET_ESP=m
CONFIG_NETWORK_SECMARK=y
CONFIG_NETFILTER=y
CONFIG_IP_NF_IPTABLES=y
CONFIG_IP_NF_FILTER=y
CONFIG_BRIDGE=m
CONFIG_BRIDGE_VLAN_FILTERING=y
CONFIG_VLAN_8021Q=m
CONFIG_NET_SCHED=y
CONFIG_NET_CLS_CGROUP=m
CONFIG_NETLINK_DIAG=y
CONFIG_NET_L3_MASTER_DEV=y
CONFIG_CGROUP_NET_PRIO=y
CONFIG_FAILOVER=y
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_MTD=y
CONFIG_MTD_BLOCK=y
CONFIG_MTD_CFI=y
CONFIG_MTD_CFI_INTELEXT=y
CONFIG_MTD_PHYSMAP=y
CONFIG_MTD_PHYSMAP_OF=y
CONFIG_BLK_DEV_LOOP=y
CONFIG_BLK_DEV_LOOP_MIN_COUNT=0
CONFIG_VIRTIO_BLK=y
CONFIG_SCSI=y
CONFIG_BLK_DEV_SD=y
CONFIG_SCSI_VIRTIO=y
CONFIG_MD=y
CONFIG_BLK_DEV_DM=y
CONFIG_NETDEVICES=y
CONFIG_MACB=y
CONFIG_PCS_XPCS=m
CONFIG_SERIO_LIBPS2=y
CONFIG_VT_HW_CONSOLE_BINDING=y
CONFIG_LEGACY_PTY_COUNT=16
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_SERIAL_OF_PLATFORM=y
CONFIG_SERIAL_EARLYCON_RISCV_SBI=y
CONFIG_VIRTIO_CONSOLE=y
CONFIG_HW_RANDOM=y
CONFIG_HW_RANDOM_VIRTIO=y
CONFIG_I2C=y
CONFIG_I2C_DESIGNWARE_CORE=y
CONFIG_SPI=y
CONFIG_PINCTRL=y
CONFIG_PINCTRL_SINGLE=y
CONFIG_GPIOLIB=y
CONFIG_GPIO_SYSFS=y
CONFIG_GPIO_DWAPB=y
CONFIG_GPIO_SIFIVE=y
CONFIG_POWER_SUPPLY=y
CONFIG_WATCHDOG=y
CONFIG_WATCHDOG_CORE=y
CONFIG_REGULATOR=y
CONFIG_REGULATOR_FIXED_VOLTAGE=y
CONFIG_BACKLIGHT_CLASS_DEVICE=m
CONFIG_SCSI_UFSHCD=y
CONFIG_SCSI_UFSHCD_PLATFORM=y
CONFIG_SCSI_UFS_DWC_TC_PLATFORM=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_DRV_M41T80=y
CONFIG_DMADEVICES=y
CONFIG_SYNC_FILE=y
CONFIG_COMMON_CLK_EYEQ=y
CONFIG_RPMSG_CHAR=y
CONFIG_RPMSG_CTRL=y
CONFIG_RPMSG_VIRTIO=y
CONFIG_RESET_CONTROLLER=y
CONFIG_RESET_SIMPLE=y
CONFIG_GENERIC_PHY=y
CONFIG_EXT4_FS=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y
CONFIG_MSDOS_FS=y
CONFIG_VFAT_FS=y
CONFIG_TMPFS=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_HUGETLBFS=y
CONFIG_KEYS=y
CONFIG_SECURITY=y
CONFIG_SECURITYFS=y
CONFIG_SECURITY_NETWORK=y
CONFIG_SECURITY_PATH=y
CONFIG_CRYPTO_RSA=y
CONFIG_CRYPTO_ECB=y
CONFIG_CRYPTO_BLAKE2B=m
CONFIG_CRYPTO_XXHASH=m
CONFIG_CRYPTO_USER_API_HASH=y
CONFIG_CRC_CCITT=m
CONFIG_CRC_ITU_T=y
CONFIG_CRC7=y
CONFIG_LIBCRC32C=m
CONFIG_PRINTK_TIME=y
CONFIG_DYNAMIC_DEBUG=y
CONFIG_DEBUG_INFO_DWARF5=y
CONFIG_DEBUG_FS=y
CONFIG_DEBUG_PAGEALLOC=y
CONFIG_PTDUMP_DEBUGFS=y
CONFIG_SCHED_STACK_END_CHECK=y
CONFIG_DEBUG_VM=y
CONFIG_DEBUG_VM_PGFLAGS=y
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_DEBUG_PER_CPU_MAPS=y
CONFIG_SOFTLOCKUP_DETECTOR=y
CONFIG_WQ_WATCHDOG=y
CONFIG_DEBUG_RT_MUTEXES=y
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_ATOMIC_SLEEP=y
CONFIG_DEBUG_LIST=y
CONFIG_DEBUG_PLIST=y
CONFIG_DEBUG_SG=y
CONFIG_RCU_EQS_DEBUG=y
CONFIG_MEMTEST=y

Quoted 5 lines
Quoted 1 line

These failures occur consistently for addresses in the 0xffffffd000000000 region.

FYI, this region is the direct mapping (see Documentation/arch/riscv/vm-layout.rst).

Thanks,

Quoted 7 lines
Quoted 2 lines
Quoted 2 lines

Alex

Hi Alex!

Do I have something to try or help to process this issue? maybe, can you give your Config and I will try it on my system? Any more information I can share?

So I'm able to reproduce your issue with your config, it only happens with kexec_load(), not kexec_file_load().

Quoted 8 lines
Quoted 7 lines

Your patch does not fix the problem for me, makedumpfile still fails. I spent quite some time looking for the code that parses the memory regions and exposes them as PT_LOAD segments in vmcore, but I did not find it, do you know where that happens for kexec_load()?

Thanks,

Alex

The code that parses memory regions is located in kexec-tools.

Damn, I was not looking there!

Quoted 2 lines

To fix the issue with missing memory regions in the vmcore, we need to ensure that kexec-tools does not exclude the Reserved-memblock sections when generating ELF headers for kexec_load().

I'm still not in favor with this solution, that does not sound right.

I think we should not exclude the "Reserved" regions which lie inside "System RAM" region. The problem is that we mark "nomap" regions as "Reserved" too, I would say this is where we are wrong: "nomap" regions don't even have direct mapping, so they should be presented as a hole rather than "Reserved". And that would allow us to not exclude the "Reserved" regions.

@Simon, @Pnina WDYT?

Thanks,

Alex

Quoted 43 lines

I’ve added a patch to handle this, and plan to submit it upstream to kexec-tools once this approach is confirmed and approved.

Kexec-tools patch:
---

kexec/arch/riscv/kexec-riscv.c +5 -3
index f34b468..1a93b51 100644
@@ -421,8 +421,11 @@ static bool to_be_excluded(char *str, unsigned long long start, unsigned long lo
  	    !strncmp(str, KERNEL_CODE, strlen(KERNEL_CODE)) ||
  	    !strncmp(str, KERNEL_DATA, strlen(KERNEL_DATA)))
  		return false;
-	else
-		return true;
+
+	if (!strncmp(str, "Reserved-memblock", strlen("Reserved-memblock")))
+		return false;
+
+	return true;
  }
 
  int get_memory_ranges(struct memory_range **range, int *num_ranges,
---

With this patch, the kexec-tools will no longer exclude the Reserved-memblock regions, allowing the crash tool to access the necessary memory areas for analysis.

Thanks,
Pnina

Quoted 13 lines
Quoted 13 lines

Thanks a lot,
Pnina

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Upon inspection, we confirmed that the physical addresses corresponding to those virtual addresses are not present in the vmcore, as they fall under Reserved memory sections. We tested a patch to kexec-tools that prevents exclusion of the Reserved-memblock section from the vmcore. With this patch, the issue no longer occurs, and crash analysis succeeds.
Note: I suspect the same issue exists on ARM64, as both the signal.c and kexec-tools implementations are similar.

Quoted 2 lines

Thanks!
Björn

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#12 lore raw

Re: [PATCH 0/1] Fix for riscv vmcore issue

Quoted 4 lines · Alexandre Ghiti

I'm still not in favor with this solution, that does not sound right.

Quoted 9 lines

I think we should not exclude the "Reserved" regions which lie inside "System RAM" region. The problem is that we mark "nomap" regions as "Reserved" too, I would say this is where we are wrong: "nomap" regions don't even have direct mapping, so they should be presented as a hole rather than "Reserved". And that would allow us to not exclude the "Reserved" regions.

@Simon, @Pnina WDYT?

Thanks,

Alex

NOMAP means the region is reserved:

https://elixir.bootlin.com/linux/v6.16-rc6/source/include/linux/memblock.h#L36

* @MEMBLOCK_NOMAP: don't add to kernel direct mapping and treat as
* reserved in the memory map; refer to memblock_mark_nomap() description
* for further details

https://elixir.bootlin.com/linux/v6.16-rc6/source/mm/memblock.c#L1060

* The memory regions marked with %MEMBLOCK_NOMAP will not be added to the
* direct mapping of the physical memory. These regions will still be
* covered by the memory map. The struct page representing NOMAP memory
* frames in the memory map will be PageReserved()
*
* Note: if the memory being marked %MEMBLOCK_NOMAP was allocated from
* memblock, the caller must inform kmemleak to ignore that memory

This is also what ARM64 does btw:

https://elixir.bootlin.com/linux/v6.16-rc6/source/arch/arm64/kernel/setup.c#L230

Sorry I didn't review this earlier...

* In the original kexec-tools port I added a function dtb_get_memory_ranges that parsed the device tree (either through /sys/firmware/fdt, or a user-provided one) for memory regions, including reserved memory regions added there e.g. by OpenSBI. This works better than using /proc/iomem, since /proc/iomem captures the memory layout of the running system, not the system we are going to boot to, and also /proc/iomem is not a standardized interface, which was yet another reason I wanted to avoid using it. I can further argue why that approach is better but it's a bit off topic here, and since we now have EFI in play it needed review anyway (when I wrote that we didn't have ACPI/EFI support, things were nice and clean). The thing is I'd prefer if there was still an option to use it, the function was upstreamed but it's not called anymore, please fix that, not everyone uses ACPI/EFI or cares about it (actually almost all RISC-V systems I've seen in production use a device tree). In our use cases where we e.g. swap accelerators on FPGAs this approach is much simpler to follow and implement, than having to re-generate ACPI tables for our target system everytime (that will have a new accelerator in place with its own reserved regions etc, that we don't want to overlap with e.g. initrd or the kernel image). Also keep in mind that kexec is not there just for kdump, it's a very useful feature for other use cases as well.

* For creating the elfcorehdr for kdump (that ends up being used for /proc/vmcore in the crashkernel), we obviously need runtime information from the running kernel (the one we want to analyze if it crashes), so the device tree couldn't obviously provide that, the standardized interface was supposed to be /proc/kcore, but that's also a security mess and is considered obsolete as far as I know (back when I was working on that I remember it was considered for removal). Which is why I used /proc/iomem and /proc/kallsyms in load_elfcorehdr like other archs do on kexec-tools, to determine the address of specific symbols to populate struct crash_elf_info, and also exclude the range of the crashkernel allocated at runtime (which I also exported via /proc/iomem). I still used the memory regions from the device tree (info->memory_ranges populated via dtb_get_memory_ranges) which meant that everything was there.

* When the code changed to rely only on /proc/iomem, although get_memory_ranges was changed, load_elfcorehdr and the regions exported via /proc/iomem remained the same. In the device tree scenario this still worked, since init_resources exposed basically the same regions as in the device tree. It should also work for the EFI scenario, by the time we reach init_resources we've already called efi_init -> reserve_regions, so we'll add both reserved and nomap regions there. For the soft-reserved regions we didn't add in efi_init, we add them later on in riscv_enable_runtime_services as "Soft Reserved" resources. I don't see why we need to add an arch initcall to run after that on resources we already added (on init_resources btw, ignoring those added by riscv_enable_runtime_services). Why do we need to do that ? Note that init_resources does two passes unlike arm64's approach, so it should handle overlapping regions properly.

Am I missing something ?

Regards,
Nick

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv