Skip to contentSkip to conversation

Handling spin table in kdump

PATCH Applied Suzuki K. Poulose 3 participants 8 messages May 22, 2012 – Jul 13, 2012

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi

I came across the following issue while testing Kdump on an SMP board(Currituck) running a non-SMP kernel. Even though the kernel is UP, the device-tree has the nodes for second CPU and the related details.

The kexec tool adds the spin table area as a reserved section in the device tree for the dump capture kernel. This value is read from the 'cpu-release-addr'.

But now, if the spin table is not located within the 'Reserved region' for the crash kernel, the dump capture kernel would fail to boot, hitting a BUG in mm/bootmem.c as in [1].

This is because we try to reserve a region which is not available to the kernel.

So I am wondering how is this handled really on an SMP board (Fsl_bookE).

There are two possible solutions :
1) Do not reserve the regions for the spin-table, as we will use only the crashing CPU in the second kernel(maxcpus=1).

2) Add the spin-table region to the available memory regions passed to the kernel by kexec-tools.

I have tested (1) and it works fine for me. Yet to test (2).

Thoughts ?

Thanks
Suzuki

[1] Kernel Bug
----------------

Linux version 3.3.0-rc5 ([email protected]) (gcc version 4.3.4 [gcc-4_3-branch revision 152973] (GCC) ) #12 Tue May 22 18:03:01 IST2 Found legacy serial port 0 for /plb/opb/serial@10000000
mem=20010000000, taddr=20010000000, irq=0, clk=1851851, speed=115200

------------[ cut here ]------------
kernel BUG at mm/bootmem.c:351!
Vector: 700 (Program Check) at [c8a61e90]
    pc: c847f91c: mark_bootmem+0xa0/0x14c
    lr: c8472670: do_init_bootmem+0x1ac/0x218
    sp: c8a61f40

msr: 21000
current = 0xc8a4a500
pid = 0, comm = swapper
kernel BUG at mm/bootmem.c:351!
enter ? for help
[c8a61f70] c8472670 do_init_bootmem+0x1ac/0x218
[c8a61f90] c847025c setup_arch+0x1bc/0x234
[c8a61fb0] c846b62c start_kernel+0x98/0x358
[c8a61ff0] c80000b4 _start+0xb4/0xf8

#1 lore raw
Quoted 25 lines · Suzuki K. Poulose

Hi

I came across the following issue while testing Kdump on an SMP board(Currituck) running a non-SMP kernel. Even though the kernel is UP, the device-tree has the nodes for second CPU and the related details.

The kexec tool adds the spin table area as a reserved section in the device tree for the dump capture kernel. This value is read from the 'cpu-release-addr'.

But now, if the spin table is not located within the 'Reserved region' for the crash kernel, the dump capture kernel would fail to boot, hitting a BUG in mm/bootmem.c as in [1].

This is because we try to reserve a region which is not available to the kernel.

So I am wondering how is this handled really on an SMP board (Fsl_bookE).

There are two possible solutions :
1) Do not reserve the regions for the spin-table, as we will use only the crashing CPU in the second kernel(maxcpus=1).

Quoted 7 lines

2) Add the spin-table region to the available memory regions passed to the kernel by kexec-tools.

I have tested (1) and it works fine for me. Yet to test (2).

Thoughts ?

I would think option 1 is acceptable. The crash kernel will never attempt to use the memory outside of the crash kernel region therefore it does not need to be reserved.

However, I thought we were able to boot SMP into the crash kernel on our parts... not sure how that effects things (although the maxcpus=1 here is a perfectly acceptable and safer thing to do anyways)

-M

Quoted 31 lines

Thanks
Suzuki

[1] Kernel Bug
----------------

Linux version 3.3.0-rc5 ([email protected]) (gcc version 4.3.4 [gcc-4_3-branch revision 152973] (GCC) ) #12 Tue May 22 18:03:01 IST2 Found legacy serial port 0 for /plb/opb/serial@10000000 mem=20010000000, taddr=20010000000, irq=0, clk=1851851, speed=115200

------------[ cut here ]------------
kernel BUG at mm/bootmem.c:351!

Vector: 700 (Program Check) at [c8a61e90]
   pc: c847f91c: mark_bootmem+0xa0/0x14c
   lr: c8472670: do_init_bootmem+0x1ac/0x218
   sp: c8a61f40
  msr: 21000
 current = 0xc8a4a500
   pid   = 0, comm = swapper
kernel BUG at mm/bootmem.c:351!
enter ? for help
[c8a61f70] c8472670 do_init_bootmem+0x1ac/0x218
[c8a61f90] c847025c setup_arch+0x1bc/0x234
[c8a61fb0] c846b62c start_kernel+0x98/0x358
[c8a61ff0] c80000b4 _start+0xb4/0xf8

Quoted 4 lines

_______________________________________________
Linuxppc-dev mailing list
[email protected]
https://lists.ozlabs.org/listinfo/linuxppc-dev

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#2 lore raw

PATCH 1/1 Do not reserve cpu spin-table for crash kernel

As of now, the kexec reserves the spin-table for all the CPUs on an SMP machine. The spin-table is pointed to by the cpu-release-addr property in the device-tree. Reserving the spin-table in the crash kernel will cause a BUG(), if the table lies outside the memory reserved for the crashkernel.

Disable reserving the spin-table regions and use maxcpus=1 to use only the crashing CPU to boot the crash kernel.

---

 kexec/arch/ppc/crashdump-powerpc.c |   19 +++++++++++++------
 kexec/arch/ppc/fixup_dtb.c         |    4 ++++
 2 files changed, 17 insertions(+), 6 deletions(-)
kexec/arch/ppc/crashdump-powerpc.c +13 -6
index 1bef69b..4c8c75d 100644
@@ -262,10 +262,19 @@ static void ulltoa(unsigned long long i, char *str)
 	}
 }
 
+/* Append str to cmdline */
+static void add_cmdline(char *cmdline, char *str)
+{
+	int cmdlen = strlen(cmdline) + strlen(str);
+	if (cmdlen > (COMMAND_LINE_SIZE - 1))
+		die("Command line overflow\n");
+	strcat(cmdline, str);
+}
+
 static int add_cmdline_param(char *cmdline, unsigned long long addr,
 				char *cmdstr, char *byte)
 {
-	int cmdlen, len, align = 1024;
+	int align = 1024;
 	char str[COMMAND_LINE_SIZE], *ptr;
 
 	/* Passing in =xxxK / =xxxM format. Saves space required in cmdline.*/
@@ -284,11 +293,8 @@ static int add_cmdline_param(char *cmdline, unsigned long long addr,
 	ptr += strlen(str);
 	ulltoa(addr, ptr);
 	strcat(str, byte);
-	len = strlen(str);
-	cmdlen = strlen(cmdline) + len;
-	if (cmdlen > (COMMAND_LINE_SIZE - 1))
-		die("Command line overflow\n");
-	strcat(cmdline, str);
+
+	add_cmdline(cmdline, str);
 
 	dbgprintf("Command line after adding elfcorehdr: %s\n", cmdline);
 
@@ -365,6 +371,7 @@ int load_crashdump_segments(struct kexec_info *info, char *mod_cmdline,
 	 */
 	add_cmdline_param(mod_cmdline, elfcorehdr, " elfcorehdr=", "K");
 	add_cmdline_param(mod_cmdline, saved_max_mem, " savemaxmem=", "M");
+	add_cmdline(mod_cmdline, " maxcpus=1");
 	return 0;
 }
 
kexec/arch/ppc/fixup_dtb.c +4 -0
index e9890a4..f832026 100644
@@ -172,6 +172,9 @@ static void fixup_reserve_regions(struct kexec_info *info, char *blob_buf)
 		}
 	}
 
+#if 0
+	/* XXX: Do not reserve spin-table for CPUs. */
+
 	/* Add reserve regions for cpu-release-addr */
 	nodeoffset = fdt_node_offset_by_prop_value(blob_buf, -1, "device_type", "cpu", 4);
 	while (nodeoffset != -FDT_ERR_NOTFOUND) {
@@ -201,6 +204,7 @@ static void fixup_reserve_regions(struct kexec_info *info, char *blob_buf)
 		nodeoffset = fdt_node_offset_by_prop_value(blob_buf, nodeoffset,
 				"device_type", "cpu", 4);
 	}
+#endif
 
 out:
 	print_fdt_reserve_regions(blob_buf);
#3 lore raw

Re: [PATCH] [ppc] Do not reserve cpu spin-table for crash kernel

Quoted 10 lines · Suzuki K. Poulose

As of now, the kexec reserves the spin-table for all the CPUs on an SMP machine. The spin-table is pointed to by the cpu-release-addr property in the device-tree. Reserving the spin-table in the crash kernel will cause a BUG(), if the table lies outside the memory reserved for the crashkernel.

Disable reserving the spin-table regions and use maxcpus=1 to use only the crashing CPU to boot the crash kernel.

Simon,

Any response on this one ?

I have tested this on a Currituck board (476, SMP) with a UP kernel. Without this patch, the secondary kernel hits 'PANIC' in boot while trying to reserve a memory(the spin table), outside the memory range(crash reserve).

Thanks
Suzuki

Quoted 82 lines

---

  kexec/arch/ppc/crashdump-powerpc.c |   19 +++++++++++++------
  kexec/arch/ppc/fixup_dtb.c         |    4 ++++
  2 files changed, 17 insertions(+), 6 deletions(-)
kexec/arch/ppc/crashdump-powerpc.c +13 -6
index 1bef69b..4c8c75d 100644
@@ -262,10 +262,19 @@ static void ulltoa(unsigned long long i, char *str)
  	}
  }
 
+/* Append str to cmdline */
+static void add_cmdline(char *cmdline, char *str)
+{
+	int cmdlen = strlen(cmdline) + strlen(str);
+	if (cmdlen>  (COMMAND_LINE_SIZE - 1))
+		die("Command line overflow\n");
+	strcat(cmdline, str);
+}
+
  static int add_cmdline_param(char *cmdline, unsigned long long addr,
  				char *cmdstr, char *byte)
  {
-	int cmdlen, len, align = 1024;
+	int align = 1024;
  	char str[COMMAND_LINE_SIZE], *ptr;
 
  	/* Passing in =xxxK / =xxxM format. Saves space required in cmdline.*/
@@ -284,11 +293,8 @@ static int add_cmdline_param(char *cmdline, unsigned long long addr,
  	ptr += strlen(str);
  	ulltoa(addr, ptr);
  	strcat(str, byte);
-	len = strlen(str);
-	cmdlen = strlen(cmdline) + len;
-	if (cmdlen>  (COMMAND_LINE_SIZE - 1))
-		die("Command line overflow\n");
-	strcat(cmdline, str);
+
+	add_cmdline(cmdline, str);
 
  	dbgprintf("Command line after adding elfcorehdr: %s\n", cmdline);
 
@@ -365,6 +371,7 @@ int load_crashdump_segments(struct kexec_info *info, char *mod_cmdline,
  	 */
  	add_cmdline_param(mod_cmdline, elfcorehdr, " elfcorehdr=", "K");
  	add_cmdline_param(mod_cmdline, saved_max_mem, " savemaxmem=", "M");
+	add_cmdline(mod_cmdline, " maxcpus=1");
  	return 0;
  }
 
kexec/arch/ppc/fixup_dtb.c +4 -0
index e9890a4..f832026 100644
@@ -172,6 +172,9 @@ static void fixup_reserve_regions(struct kexec_info *info, char *blob_buf)
  		}
  	}
 
+#if 0
+	/* XXX: Do not reserve spin-table for CPUs. */
+
  	/* Add reserve regions for cpu-release-addr */
  	nodeoffset = fdt_node_offset_by_prop_value(blob_buf, -1, "device_type", "cpu", 4);
  	while (nodeoffset != -FDT_ERR_NOTFOUND) {
@@ -201,6 +204,7 @@ static void fixup_reserve_regions(struct kexec_info *info, char *blob_buf)
  		nodeoffset = fdt_node_offset_by_prop_value(blob_buf, nodeoffset,
  				"device_type", "cpu", 4);
  	}
+#endif
 
  out:
  	print_fdt_reserve_regions(blob_buf);

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#4 lore raw

Re: [PATCH] [ppc] Do not reserve cpu spin-table for crash kernel

Simon,

Please let me know your response.

Thanks
Suzuki

Quoted 108 lines · Suzuki K. Poulose
Quoted 10 lines · Suzuki K. Poulose

As of now, the kexec reserves the spin-table for all the CPUs on an SMP machine. The spin-table is pointed to by the cpu-release-addr property in the device-tree. Reserving the spin-table in the crash kernel will cause a BUG(), if the table lies outside the memory reserved for the crashkernel.

Disable reserving the spin-table regions and use maxcpus=1 to use only the crashing CPU to boot the crash kernel.

Simon,

Any response on this one ?

I have tested this on a Currituck board (476, SMP) with a UP kernel. Without this patch, the secondary kernel hits 'PANIC' in boot while trying to reserve a memory(the spin table), outside the memory range(crash reserve).

Thanks
Suzuki

Quoted 83 lines

---

kexec/arch/ppc/crashdump-powerpc.c | 19 +++++++++++++------ kexec/arch/ppc/fixup_dtb.c | 4 ++++

2 files changed, 17 insertions(+), 6 deletions(-)
kexec/arch/ppc/crashdump-powerpc.c +13 -6
index 1bef69b..4c8c75d 100644
@@ -262,10 +262,19 @@ static void ulltoa(unsigned long long i, char *str)
 }
 }
 
+/* Append str to cmdline */
+static void add_cmdline(char *cmdline, char *str)
+{
+ int cmdlen = strlen(cmdline) + strlen(str);
+ if (cmdlen> (COMMAND_LINE_SIZE - 1))
+ die("Command line overflow\n");
+ strcat(cmdline, str);
+}
+
 static int add_cmdline_param(char *cmdline, unsigned long long addr,
 char *cmdstr, char *byte)
 {
- int cmdlen, len, align = 1024;
+ int align = 1024;
 char str[COMMAND_LINE_SIZE], *ptr;
 
 /* Passing in =xxxK / =xxxM format. Saves space required in cmdline.*/
@@ -284,11 +293,8 @@ static int add_cmdline_param(char *cmdline,
 unsigned long long addr,
 ptr += strlen(str);
 ulltoa(addr, ptr);
 strcat(str, byte);
- len = strlen(str);
- cmdlen = strlen(cmdline) + len;
- if (cmdlen> (COMMAND_LINE_SIZE - 1))
- die("Command line overflow\n");
- strcat(cmdline, str);
+
+ add_cmdline(cmdline, str);
 
 dbgprintf("Command line after adding elfcorehdr: %s\n", cmdline);
@@ -365,6 +371,7 @@ int load_crashdump_segments(struct kexec_info
 *info, char *mod_cmdline,
 */
 add_cmdline_param(mod_cmdline, elfcorehdr, " elfcorehdr=", "K");
 add_cmdline_param(mod_cmdline, saved_max_mem, " savemaxmem=", "M");
+ add_cmdline(mod_cmdline, " maxcpus=1");
 return 0;
 }
kexec/arch/ppc/fixup_dtb.c +3 -0
index e9890a4..f832026 100644
@@ -172,6 +172,9 @@ static void fixup_reserve_regions(struct
 kexec_info *info, char *blob_buf)
 }
 }
 
+#if 0
+ /* XXX: Do not reserve spin-table for CPUs. */
+
 /* Add reserve regions for cpu-release-addr */
 nodeoffset = fdt_node_offset_by_prop_value(blob_buf, -1,

"device_type", "cpu", 4);
while (nodeoffset != -FDT_ERR_NOTFOUND) {

kexec/arch/ppc/fixup_dtb.c +1 -0
@@ -201,6 +204,7 @@ static void fixup_reserve_regions(struct
 kexec_info *info, char *blob_buf)
 nodeoffset = fdt_node_offset_by_prop_value(blob_buf, nodeoffset,
 "device_type", "cpu", 4);
 }
+#endif
 

out:
print_fdt_reserve_regions(blob_buf);

Quoted 12 lines
Quoted 5 lines

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#5 lore raw

Re: [PATCH] [ppc] Do not reserve cpu spin-table for crash kernel

Quoted 2 lines · Suzuki K. Poulose

Simon,

Ping.

Suzuki

Quoted 129 lines

Please let me know your response.

Thanks
Suzuki

Quoted 109 lines · Suzuki K. Poulose
Quoted 10 lines · Suzuki K. Poulose

As of now, the kexec reserves the spin-table for all the CPUs on an SMP machine. The spin-table is pointed to by the cpu-release-addr property in the device-tree. Reserving the spin-table in the crash kernel will cause a BUG(), if the table lies outside the memory reserved for the crashkernel.

Disable reserving the spin-table regions and use maxcpus=1 to use only the crashing CPU to boot the crash kernel.

Simon,

Any response on this one ?

I have tested this on a Currituck board (476, SMP) with a UP kernel. Without this patch, the secondary kernel hits 'PANIC' in boot while trying to reserve a memory(the spin table), outside the memory range(crash reserve).

Thanks
Suzuki

Quoted 84 lines

---

kexec/arch/ppc/crashdump-powerpc.c | 19 +++++++++++++------ kexec/arch/ppc/fixup_dtb.c | 4 ++++

2 files changed, 17 insertions(+), 6 deletions(-)
kexec/arch/ppc/crashdump-powerpc.c +10 -1
index 1bef69b..4c8c75d 100644
@@ -262,10 +262,19 @@ static void ulltoa(unsigned long long i, char
 *str)
 }
 }
 
+/* Append str to cmdline */
+static void add_cmdline(char *cmdline, char *str)
+{
+ int cmdlen = strlen(cmdline) + strlen(str);
+ if (cmdlen> (COMMAND_LINE_SIZE - 1))
+ die("Command line overflow\n");
+ strcat(cmdline, str);
+}
+
 static int add_cmdline_param(char *cmdline, unsigned long long addr,
 char *cmdstr, char *byte)
 {
- int cmdlen, len, align = 1024;
+ int align = 1024;
 char str[COMMAND_LINE_SIZE], *ptr;
 

/* Passing in =xxxK / =xxxM format. Saves space required in cmdline.*/

kexec/arch/ppc/crashdump-powerpc.c +3 -5
@@ -284,11 +293,8 @@ static int add_cmdline_param(char *cmdline,
 unsigned long long addr,
 ptr += strlen(str);
 ulltoa(addr, ptr);
 strcat(str, byte);
- len = strlen(str);
- cmdlen = strlen(cmdline) + len;
- if (cmdlen> (COMMAND_LINE_SIZE - 1))
- die("Command line overflow\n");
- strcat(cmdline, str);
+
+ add_cmdline(cmdline, str);
 
 dbgprintf("Command line after adding elfcorehdr: %s\n", cmdline);
@@ -365,6 +371,7 @@ int load_crashdump_segments(struct kexec_info
 *info, char *mod_cmdline,
 */
 add_cmdline_param(mod_cmdline, elfcorehdr, " elfcorehdr=", "K");
 add_cmdline_param(mod_cmdline, saved_max_mem, " savemaxmem=", "M");
+ add_cmdline(mod_cmdline, " maxcpus=1");
 return 0;
 }
kexec/arch/ppc/fixup_dtb.c +3 -0
index e9890a4..f832026 100644
@@ -172,6 +172,9 @@ static void fixup_reserve_regions(struct
 kexec_info *info, char *blob_buf)
 }
 }
 
+#if 0
+ /* XXX: Do not reserve spin-table for CPUs. */
+
 /* Add reserve regions for cpu-release-addr */
 nodeoffset = fdt_node_offset_by_prop_value(blob_buf, -1,

"device_type", "cpu", 4);
while (nodeoffset != -FDT_ERR_NOTFOUND) {

kexec/arch/ppc/fixup_dtb.c +1 -0
@@ -201,6 +204,7 @@ static void fixup_reserve_regions(struct
 kexec_info *info, char *blob_buf)
 nodeoffset = fdt_node_offset_by_prop_value(blob_buf, nodeoffset,
 "device_type", "cpu", 4);
 }
+#endif
 

out:
print_fdt_reserve_regions(blob_buf);

Quoted 12 lines
Quoted 5 lines

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

Quoted 5 lines

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

Applied
#7 lore raw

Re: [PATCH] [ppc] Do not reserve cpu spin-table for crash kernel

In reply to Suzuki K. Poulose
Quoted 10 lines · Suzuki K. Poulose

As of now, the kexec reserves the spin-table for all the CPUs on an SMP machine. The spin-table is pointed to by the cpu-release-addr property in the device-tree. Reserving the spin-table in the crash kernel will cause a BUG(), if the table lies outside the memory reserved for the crashkernel.

Disable reserving the spin-table regions and use maxcpus=1 to use only the crashing CPU to boot the crash kernel.

Thanks, applied.

Sorry for not seeing this (much) earlier.

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec