Skip to contentSkip to conversation

kdump: quad core Opteron

DISCUSSION Chandru 5 participants 13 messages Oct 7, 2008 – Dec 9, 2008

Subsystems (2)
lore
Share & tools
lore

Chandru

author
#0 lore raw

kdump on a quad core Opteron blade machine doesn't give a complete
vmcore on the system. All works well until we attempt to copy
/proc/vmcore to some target place ( disk , n/w ). The system immediately resets without any OS messages after having copied few mb's of vmcore file. Problem also occurs with 2.6.27-rc8 and latest kexec-tools. If we pass 'mem=4G' as boot parameter to the first kernel, then kdump succeeds in copying a readable vmcore to /var/crash.

[root@abc ~]# uname -a
Linux abc 2.6.27-rc8 #2 SMP Tue Oct 7 08:05:46 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux

Chandru

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#1 lore raw
Quoted 8 lines · Chandru

kdump on a quad core Opteron blade machine doesn't give a complete
vmcore on the system. All works well until we attempt to copy
/proc/vmcore to some target place ( disk , n/w ). The system immediately resets without any OS messages after having copied few mb's of vmcore file. Problem also occurs with 2.6.27-rc8 and latest kexec-tools. If we pass 'mem=4G' as boot parameter to the first kernel, then kdump succeeds in copying a readable vmcore to /var/crash.

Maybe extend the copy process to print out the address, and then look at which address it crashes. Then look in the memory map, maybe there's something special there.

Regards,
Bernhard

Signature · 6 lines
Bernhard Walle, SUSE Linux Products GmbH, Architecture Development

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#2 lore raw
In reply to Chandru
Quoted 8 lines · Chandru

kdump on a quad core Opteron blade machine doesn't give a complete
vmcore on the system. All works well until we attempt to copy
/proc/vmcore to some target place ( disk , n/w ). The system immediately resets without any OS messages after having copied few mb's of vmcore file. Problem also occurs with 2.6.27-rc8 and latest kexec-tools. If we pass 'mem=4G' as boot parameter to the first kernel, then kdump succeeds in copying a readable vmcore to /var/crash.

Hi Chandru,

How much memory this system has got. Can you also paste the output of /proc/iomem of first kernel.

Does this system has GART? So looks like we are accessing some memory area which platform does not like. (We saw issues with GART in the past.)

Can you also provide /proc/vmcore ELF header (readelf output), in both the cases (mem=4G and without that).

You can try putting some printk in /proc/vmcore code and see which physical memory area you are accessing when system goes bust. If in all the failure cases it is same physical memory area, then we can try to find what's so special about it.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#3 lore raw
Quoted 18 lines · Vivek Goyal
Quoted 8 lines · Chandru

kdump on a quad core Opteron blade machine doesn't give a complete
vmcore on the system. All works well until we attempt to copy
/proc/vmcore to some target place ( disk , n/w ). The system immediately resets without any OS messages after having copied few mb's of vmcore file. Problem also occurs with 2.6.27-rc8 and latest kexec-tools. If we pass 'mem=4G' as boot parameter to the first kernel, then kdump succeeds in copying a readable vmcore to /var/crash.

Hi Chandru,

How much memory this system has got. Can you also paste the output of /proc/iomem of first kernel.

Does this system has GART? So looks like we are accessing some memory area which platform does not like. (We saw issues with GART in the past.)

Quoted 7 lines

Can you also provide /proc/vmcore ELF header (readelf output), in both the cases (mem=4G and without that).

You can try putting some printk in /proc/vmcore code and see which physical memory area you are accessing when system goes bust. If in all the failure cases it is same physical memory area, then we can try to find what's so special about it.

Or you can assume this is pretty much exactly the problem I ran into in August. I've attached the patch that I'm using with our 2.6.18 kernel to disable CPU-side access by the GART, which prevents the problem on our Family 10H systems. You'll need to fix the directory name for kernels newer than the arch/x86_64 merge.

Now that someone else has seen the problem, if this fixes it, I'll submit the patch upstream.

Here's the README for the patch:

This patch changes the initialization of the GART (in pci-gart.c:init_k8_gatt) to set the DisGartCpu bit in the GART Aperture Control Register. Setting the bit Disables requests from the CPUs from accessing the GART. In other words, CPU memory accesses within the range of addresses in the aperture will not cause the GART to perform an address translation. The aperture area was already being unmapped at the kernel level with clear_kernel_mapping() to prevent accesses from the CPU, but that kernel level unmapping is not in effect in the kexec'd kdump kernel. By disabling the CPU-side accesses within the GART, which does persist through the kexec of the kdump kernel, the kdump kernel is prevented from interacting with the GART during accesses to the dump memory areas which include the address range of the GART aperture. Although the patch can be applied to the kdump kernel, it is not exercised there because the kdump kernel doesn't attempt to initialize the GART.

Bob Montgomery
working at HP

arch/x86_64/kernel/pci-gart.c renamed from arch/x86_64/kernel/pci-gart.c.orig +3 -2
@@ -540,8 +540,9 @@ static __init int init_k8_gatt(struct ag
 		pci_write_config_dword(dev, 0x98, gatt_reg);
 		pci_read_config_dword(dev, 0x90, &ctl); 
 
-		ctl |= 1;
-		ctl &= ~((1<<4) | (1<<5));
+		ctl |= 1;	/* set GartEn */
+		ctl |= (1<<4);	/* set DisGartCpu */
+		ctl &= ~(1<<5); /* clear DisGartIO */
 
 		pci_write_config_dword(dev, 0x90, ctl); 
 	}

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

author
#4 lore raw
Quoted 18 lines · Bob Montgomery

Here's the README for the patch:

This patch changes the initialization of the GART (in pci-gart.c:init_k8_gatt) to set the DisGartCpu bit in the GART Aperture Control Register. Setting the bit Disables requests from the CPUs from accessing the GART. In other words, CPU memory accesses within the range of addresses in the aperture will not cause the GART to perform an address translation. The aperture area was already being unmapped at the kernel level with clear_kernel_mapping() to prevent accesses from the CPU, but that kernel level unmapping is not in effect in the kexec'd kdump kernel. By disabling the CPU-side accesses within the GART, which does persist through the kexec of the kdump kernel, the kdump kernel is prevented from interacting with the GART during accesses to the dump memory areas which include the address range of the GART aperture. Although the patch can be applied to the kdump kernel, it is not exercised there because the kdump kernel doesn't attempt to initialize the GART.

Quoted 4 lines

Bob Montgomery
working at HP

I made the following changes and it seems like it is working.

--- linux-2.6.27-rc8/include/asm-x86/gart.h.orig 2008-10-08
09:41:23.000000000 -0400
+++ linux-2.6.27-rc8/include/asm-x86/gart.h 2008-10-08
09:44:35.000000000 -0400
@@ -42,7 +42,8 @@ static inline void enable_gart_translati

         /* Enable GART translation for this hammer. */
         pci_read_config_dword(dev, AMD64_GARTAPERTURECTL, &ctl);
         ctl |= GARTEN;
-        ctl &= ~(DISGARTCPU | DISGARTIO);
+       ctl |= DISGARTCPU;
+       ctl &= ~(DISGARTIO);
         pci_write_config_dword(dev, AMD64_GARTAPERTURECTL, ctl);

}

The system doesn't reset now while copying /proc/vmcore. Thanks!

Chandru

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

author
#5 lore raw
In reply to Bob Montgomery
Quoted 53 lines · On Tuesday 07 October 2008 21:29:51 Bob Montgomery wrote:
Quoted 17 lines · Vivek Goyal
Quoted 7 lines · Chandru

kdump on a quad core Opteron blade machine doesn't give a complete
vmcore on the system. All works well until we attempt to copy
/proc/vmcore to some target place ( disk , n/w ). The system immediately resets without any OS messages after having copied few mb's of vmcore file. Problem also occurs with 2.6.27-rc8 and latest kexec-tools. If we pass 'mem=4G' as boot parameter to the first kernel, then kdump succeeds in copying a readable vmcore to /var/crash.

Hi Chandru,

How much memory this system has got. Can you also paste the output of /proc/iomem of first kernel.

Does this system has GART? So looks like we are accessing some memory area which platform does not like. (We saw issues with GART in the past.)

Quoted 7 lines

Can you also provide /proc/vmcore ELF header (readelf output), in both the cases (mem=4G and without that).

You can try putting some printk in /proc/vmcore code and see which physical memory area you are accessing when system goes bust. If in all the failure cases it is same physical memory area, then we can try to find what's so special about it.

Or you can assume this is pretty much exactly the problem I ran into in August. I've attached the patch that I'm using with our 2.6.18 kernel to disable CPU-side access by the GART, which prevents the problem on our Family 10H systems. You'll need to fix the directory name for kernels newer than the arch/x86_64 merge.

Now that someone else has seen the problem, if this fixes it, I'll submit the patch upstream.

Here's the README for the patch:

This patch changes the initialization of the GART (in pci-gart.c:init_k8_gatt) to set the DisGartCpu bit in the GART Aperture Control Register. Setting the bit Disables requests from the CPUs from accessing the GART. In other words, CPU memory accesses within the range of addresses in the aperture will not cause the GART to perform an address translation. The aperture area was already being unmapped at the kernel level with clear_kernel_mapping() to prevent accesses from the CPU, but that kernel level unmapping is not in effect in the kexec'd kdump kernel. By disabling the CPU-side accesses within the GART, which does persist through the kexec of the kdump kernel, the kdump kernel is prevented from interacting with the GART during accesses to the dump memory areas which include the address range of the GART aperture. Although the patch can be applied to the kdump kernel, it is not exercised there because the kdump kernel doesn't attempt to initialize the GART.

Quoted 2 lines

Bob Montgomery
working at HP

Hi Bob,

This problem was recently reported on a LS42 blade and the patch given by you also resolved the issue here too. However I made couple of changes to kexec-tools to ignore GART memory region and not have elf headers created to it. This patch also seemed to work on a LS21.

Thanks,
Chandru

---

--- kexec-tools/kexec/arch/x86_64/crashdump-x86_64.c.orig 2008-12-08
01:50:41.000000000 -0600
+++ kexec-tools/kexec/arch/x86_64/crashdump-x86_64.c 2008-12-08
03:02:45.000000000 -0600
@@ -47,7 +47,7 @@ static struct crash_elf_info elf_info =
};

/* Forward Declaration. */
-static int exclude_crash_reserve_region(int *nr_ranges); +static int exclude_region(int *nr_ranges, uint64_t start, uint64_t end);

#define KERN_VADDR_ALIGN 0x100000 /* 1MB */

@@ -164,10 +164,11 @@ static struct memory_range crash_reserve
static int get_crash_memory_ranges(struct memory_range **range, int *ranges) {
const char *iomem= proc_iomem();
- int memory_ranges = 0;
+ int memory_ranges = 0, gart = 0;
char line[MAX_LINE];
FILE *fp;
unsigned long long start, end;
+ uint64_t gart_start = 0, gart_end = 0;

fp = fopen(iomem, "r");
if (!fp) {
@@ -219,6 +220,10 @@ static int get_crash_memory_ranges(struc
type = RANGE_ACPI;
} else if(memcmp(str,"ACPI Non-volatile Storage\n",26) == 0 ) {
type = RANGE_ACPI_NVS;
+ } else if (memcmp(str, "GART\n", 5) == 0) {
+ gart_start = start;
+ gart_end = end;
+ gart = 1;
} else {
continue;
}
@@ -233,8 +238,14 @@ static int get_crash_memory_ranges(struc
memory_ranges++;
}
fclose(fp);
- if (exclude_crash_reserve_region(&memory_ranges) < 0)
+ if (exclude_region(&memory_ranges, crash_reserved_mem.start,
+ crash_reserved_mem.end) < 0)
return -1;
+ if (gart) {
+ /* exclude GART region if the system has one */
+ if (exclude_region(&memory_ranges, gart_start, gart_end) < 0)
+ return -1;
+ }
*range = crash_memory_range;
*ranges = memory_ranges;
#ifdef DEBUG
@@ -252,32 +263,27 @@ static int get_crash_memory_ranges(struc
/* Removes crash reserve region from list of memory chunks for whom elf
program
* headers have to be created. Assuming crash reserve region to be a single
* continuous area fully contained inside one of the memory chunks */
-static int exclude_crash_reserve_region(int *nr_ranges)
+static int exclude_region(int *nr_ranges, uint64_t start, uint64_t end)
{
int i, j, tidx = -1;
- unsigned long long cstart, cend;
struct memory_range temp_region;

- /* Crash reserved region. */
- cstart = crash_reserved_mem.start;
- cend = crash_reserved_mem.end;
-
for (i = 0; i < (*nr_ranges); i++) {
unsigned long long mstart, mend; mstart = crash_memory_range[i].start; mend = crash_memory_range[i].end;
- if (cstart < mend && cend > mstart) {
- if (cstart != mstart && cend != mend) {
+ if (start < mend && end > mstart) {
+ if (start != mstart && end != mend) {
/* Split memory region */
- crash_memory_range[i].end = cstart - 1;
- temp_region.start = cend + 1;
+ crash_memory_range[i].end = start - 1;
+ temp_region.start = end + 1;
temp_region.end = mend; temp_region.type = RANGE_RAM; tidx = i+1;
- } else if (cstart != mstart)
- crash_memory_range[i].end = cstart - 1;
+ } else if (start != mstart)
+ crash_memory_range[i].end = start - 1;
else
- crash_memory_range[i].start = cend + 1;
+ crash_memory_range[i].start = end + 1;
}
}
/* Insert split memory region, if any. */

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#6 lore raw
Quoted 64 lines · Chandru
Quoted 53 lines · On Tuesday 07 October 2008 21:29:51 Bob Montgomery wrote:
Quoted 17 lines · Vivek Goyal
Quoted 7 lines · Chandru

kdump on a quad core Opteron blade machine doesn't give a complete
vmcore on the system. All works well until we attempt to copy
/proc/vmcore to some target place ( disk , n/w ). The system immediately resets without any OS messages after having copied few mb's of vmcore file. Problem also occurs with 2.6.27-rc8 and latest kexec-tools. If we pass 'mem=4G' as boot parameter to the first kernel, then kdump succeeds in copying a readable vmcore to /var/crash.

Hi Chandru,

How much memory this system has got. Can you also paste the output of /proc/iomem of first kernel.

Does this system has GART? So looks like we are accessing some memory area which platform does not like. (We saw issues with GART in the past.)

Quoted 7 lines

Can you also provide /proc/vmcore ELF header (readelf output), in both the cases (mem=4G and without that).

You can try putting some printk in /proc/vmcore code and see which physical memory area you are accessing when system goes bust. If in all the failure cases it is same physical memory area, then we can try to find what's so special about it.

Or you can assume this is pretty much exactly the problem I ran into in August. I've attached the patch that I'm using with our 2.6.18 kernel to disable CPU-side access by the GART, which prevents the problem on our Family 10H systems. You'll need to fix the directory name for kernels newer than the arch/x86_64 merge.

Now that someone else has seen the problem, if this fixes it, I'll submit the patch upstream.

Here's the README for the patch:

This patch changes the initialization of the GART (in pci-gart.c:init_k8_gatt) to set the DisGartCpu bit in the GART Aperture Control Register. Setting the bit Disables requests from the CPUs from accessing the GART. In other words, CPU memory accesses within the range of addresses in the aperture will not cause the GART to perform an address translation. The aperture area was already being unmapped at the kernel level with clear_kernel_mapping() to prevent accesses from the CPU, but that kernel level unmapping is not in effect in the kexec'd kdump kernel. By disabling the CPU-side accesses within the GART, which does persist through the kexec of the kdump kernel, the kdump kernel is prevented from interacting with the GART during accesses to the dump memory areas which include the address range of the GART aperture. Although the patch can be applied to the kdump kernel, it is not exercised there because the kdump kernel doesn't attempt to initialize the GART.

Quoted 2 lines

Bob Montgomery
working at HP

Hi Bob,

This problem was recently reported on a LS42 blade and the patch given by you also resolved the issue here too. However I made couple of changes to kexec-tools to ignore GART memory region and not have elf headers created to it. This patch also seemed to work on a LS21.

Quoted 5 lines

Thanks,
Chandru

---

Hi Chandru,

So this patch will solve the issue (at least for /proc/vmcore) even if we don't make any changes on kernel side?

Thanks
Vivek

Quoted 86 lines

--- kexec-tools/kexec/arch/x86_64/crashdump-x86_64.c.orig 2008-12-08
01:50:41.000000000 -0600
+++ kexec-tools/kexec/arch/x86_64/crashdump-x86_64.c 2008-12-08
03:02:45.000000000 -0600

(quoted diff) +14 -3
@@ -47,7 +47,7 @@ static struct crash_elf_info elf_info =
 };
 
 /* Forward Declaration. */
-static int exclude_crash_reserve_region(int *nr_ranges);
+static int exclude_region(int *nr_ranges, uint64_t start, uint64_t end);
 
 #define KERN_VADDR_ALIGN	0x100000	/* 1MB */
 
@@ -164,10 +164,11 @@ static struct memory_range crash_reserve
 static int get_crash_memory_ranges(struct memory_range **range, int *ranges)
 {
 	const char *iomem= proc_iomem();
-	int memory_ranges = 0;
+	int memory_ranges = 0, gart = 0;
 	char line[MAX_LINE];
 	FILE *fp;
 	unsigned long long start, end;
+	uint64_t gart_start = 0, gart_end = 0;
 
 	fp = fopen(iomem, "r");
 	if (!fp) {
@@ -219,6 +220,10 @@ static int get_crash_memory_ranges(struc
 			type = RANGE_ACPI;
 		} else if(memcmp(str,"ACPI Non-volatile Storage\n",26) == 0 ) {
 			type = RANGE_ACPI_NVS;
+		} else if (memcmp(str, "GART\n", 5) == 0) {
+			gart_start = start;
+			gart_end = end;
+			gart = 1;
 		} else {
 			continue;
 		}
@@ -233,8 +238,14 @@ static int get_crash_memory_ranges(struc
 		memory_ranges++;
 	}
 	fclose(fp);
-	if (exclude_crash_reserve_region(&memory_ranges) < 0)
+	if (exclude_region(&memory_ranges, crash_reserved_mem.start,
+				crash_reserved_mem.end) < 0)
 		return -1;
+	if (gart) {
+		/* exclude GART region if the system has one */
+		if (exclude_region(&memory_ranges, gart_start, gart_end) < 0)
+			return -1;
+	}
 	*range = crash_memory_range;
 	*ranges = memory_ranges;
 #ifdef DEBUG
@@ -252,32 +263,27 @@ static int get_crash_memory_ranges(struc
 /* Removes crash reserve region from list of memory chunks for whom elf 

program
* headers have to be created. Assuming crash reserve region to be a single
* continuous area fully contained inside one of the memory chunks */

(quoted diff) +5 -10
-static int exclude_crash_reserve_region(int *nr_ranges)
+static int exclude_region(int *nr_ranges, uint64_t start, uint64_t end)
 {
 	int i, j, tidx = -1;
-	unsigned long long cstart, cend;
 	struct memory_range temp_region;
 
-	/* Crash reserved region. */
-	cstart = crash_reserved_mem.start;
-	cend = crash_reserved_mem.end;
-
 	for (i = 0; i < (*nr_ranges); i++) {
 		unsigned long long mstart, mend;
 		mstart = crash_memory_range[i].start;
 		mend = crash_memory_range[i].end;
-		if (cstart < mend && cend > mstart) {
-			if (cstart != mstart && cend != mend) {
+		if (start < mend && end > mstart) {
+			if (start != mstart && end != mend) {
 				/* Split memory region */
-				crash_memory_range[i].end = cstart - 1;
-				temp_region.start = cend + 1;
+				crash_memory_range[i].end = start - 1;
+				temp_region.start = end + 1;
 				temp_region.end = mend;
 				temp_region.type = RANGE_RAM;
 				tidx = i+1;

Review

Quoted 10 lines
(quoted diff) +3 -3
-			} else if (cstart != mstart)
-				crash_memory_range[i].end = cstart - 1;
+			} else if (start != mstart)
+				crash_memory_range[i].end = start - 1;
 			else
-				crash_memory_range[i].start = cend + 1;
+				crash_memory_range[i].start = end + 1;
 		}
 	}
 	/* Insert split memory region, if any. */

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

author
#7 lore raw
Quoted 7 lines · On Tuesday 09 December 2008 03:24:32 Vivek Goyal wrote:

Hi Chandru,

So this patch will solve the issue (at least for /proc/vmcore) even if we don't make any changes on kernel side?

Thanks
Vivek

Hi Vivek,

Yes, without any changes to the kernel and with this patch, we can boot into kdump kernel on these quad core Opteron machines and collect a dump.

thanks,
Chandru

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#8 lore raw
Quoted 7 lines · Chandru
Quoted 5 lines · On Tuesday 09 December 2008 03:24:32 Vivek Goyal wrote:

Hi Chandru,

So this patch will solve the issue (at least for /proc/vmcore) even if we don't make any changes on kernel side?

Quoted 7 lines
Quoted 2 lines

Thanks
Vivek

Hi Vivek,

Yes, without any changes to the kernel and with this patch, we can boot into kdump kernel on these quad core Opteron machines and collect a dump.

Provided that the kernel is new enough to have the "Insert GART region into resource map" already. Otherwise, you will need to change the kernel for this to work.

Chandru, thanks for coding this up.

Bob Montgomery

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#9 lore raw
In reply to Chandru
Quoted 9 lines · Chandru

Hi Bob,

This problem was recently reported on a LS42 blade and the patch given by you also resolved the issue here too. However I made couple of changes to kexec-tools to ignore GART memory region and not have elf headers created to it. This patch also seemed to work on a LS21.

Thanks,
Chandru

Hi Chandru,

I tried your patch on kexec-tools, and I'm seeing a zero-length section in /proc/vmcore (using readelf -e /proc/vmcore) right after the GART hole:

/proc/iomem in the main kernel shows:
01000000-08ffffff : Crash kernel
20000000-23ffffff : GART
cfe4e000-cfe55fff : ACPI Tables

And readelf -e /proc/vmcore in the kdump kernel shows:

Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
...
LOAD 0x000000000144d99c 0xffff810009000000 0x0000000009000000

                 0x0000000017000000 0x0000000017000000  RWE    0
  LOAD           0x000000001844d99c 0xffff810024000000 0x0000000024000000
                 0x00000000abe4e000 0x00000000abe4e000  RWE    0
  LOAD           0x00000000c429b99c 0xffff810024000000 0x0000000024000000
                 0x0000000000000000 0x0000000000000000  RWE    0

...

The first LOAD shown covers 09000000-20000000, the System RAM between the Crash kernel and the GART.
The next LOAD covers 24000000-cfe4e000, which is the System RAM between the GART and the ACPI Tables. So that all looks good.

Then the next LOAD is also at 24000000 with a 0 in the size fields. I haven't had a chance to check the code yet.

Bob Montgomery

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#10 lore raw
Quoted 42 lines · Bob Montgomery
Quoted 9 lines · Chandru

Hi Bob,

This problem was recently reported on a LS42 blade and the patch given by you also resolved the issue here too. However I made couple of changes to kexec-tools to ignore GART memory region and not have elf headers created to it. This patch also seemed to work on a LS21.

Thanks,
Chandru

Hi Chandru,

I tried your patch on kexec-tools, and I'm seeing a zero-length section in /proc/vmcore (using readelf -e /proc/vmcore) right after the GART hole:

/proc/iomem in the main kernel shows:
01000000-08ffffff : Crash kernel
20000000-23ffffff : GART
cfe4e000-cfe55fff : ACPI Tables

And readelf -e /proc/vmcore in the kdump kernel shows:

Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
...
LOAD 0x000000000144d99c 0xffff810009000000 0x0000000009000000

                 0x0000000017000000 0x0000000017000000  RWE    0
  LOAD           0x000000001844d99c 0xffff810024000000 0x0000000024000000
                 0x00000000abe4e000 0x00000000abe4e000  RWE    0
  LOAD           0x00000000c429b99c 0xffff810024000000 0x0000000024000000
                 0x0000000000000000 0x0000000000000000  RWE    0

...

The first LOAD shown covers 09000000-20000000, the System RAM between the Crash kernel and the GART.
The next LOAD covers 24000000-cfe4e000, which is the System RAM between the GART and the ACPI Tables. So that all looks good.

Quoted 5 lines

Then the next LOAD is also at 24000000 with a 0 in the size fields. I haven't had a chance to check the code yet.

Bob Montgomery

Seems like something we can work out pretty easily. Did crash choke on this at all?

Neil

Quoted 6 lines

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

Signature · 13 lines
/***************************************************
 *Neil Horman
 *Senior Software Engineer
 *Red Hat, Inc.
 *[email protected]
 *gpg keyid: 1024D / 0x92A74FA1
 *http://pgp.mit.edu
 ***************************************************/

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
author
#11 lore raw
In reply to Bob Montgomery
Quoted 41 lines · On Tuesday 09 December 2008 05:05:16 Bob Montgomery wrote:
Quoted 9 lines · Chandru

Hi Bob,

This problem was recently reported on a LS42 blade and the patch given by you also resolved the issue here too. However I made couple of changes to kexec-tools to ignore GART memory region and not have elf headers created to it. This patch also seemed to work on a LS21.

Thanks,
Chandru

Hi Chandru,

I tried your patch on kexec-tools, and I'm seeing a zero-length section in /proc/vmcore (using readelf -e /proc/vmcore) right after the GART hole:

/proc/iomem in the main kernel shows:
01000000-08ffffff : Crash kernel
20000000-23ffffff : GART
cfe4e000-cfe55fff : ACPI Tables

And readelf -e /proc/vmcore in the kdump kernel shows:

Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
...
LOAD 0x000000000144d99c 0xffff810009000000 0x0000000009000000

                 0x0000000017000000 0x0000000017000000  RWE    0
  LOAD           0x000000001844d99c 0xffff810024000000 0x0000000024000000
                 0x00000000abe4e000 0x00000000abe4e000  RWE    0
  LOAD           0x00000000c429b99c 0xffff810024000000 0x0000000024000000
                 0x0000000000000000 0x0000000000000000  RWE    0

...

The first LOAD shown covers 09000000-20000000, the System RAM between the Crash kernel and the GART.
The next LOAD covers 24000000-cfe4e000, which is the System RAM between the GART and the ACPI Tables. So that all looks good.

Quoted 4 lines

Then the next LOAD is also at 24000000 with a 0 in the size fields. I haven't had a chance to check the code yet.

Bob Montgomery

Whoops, a 'continue' was missing in the patch. Here is the updated patch...

Exclude GART memory region and make kexec-tools to not create elf headers to it. Currently it seems like the dump analysis tools do not need a copy of the GART memory region, hence ignoring it in kexec-tools.

---

--- kexec-tools/kexec/arch/x86_64/crashdump-x86_64.c.orig 2008-12-08
01:50:41.000000000 -0600
+++ kexec-tools/kexec/arch/x86_64/crashdump-x86_64.c 2008-12-08
22:01:50.000000000 -0600
@@ -47,7 +47,7 @@ static struct crash_elf_info elf_info =
};

/* Forward Declaration. */
-static int exclude_crash_reserve_region(int *nr_ranges); +static int exclude_region(int *nr_ranges, uint64_t start, uint64_t end);

#define KERN_VADDR_ALIGN 0x100000 /* 1MB */

@@ -164,10 +164,11 @@ static struct memory_range crash_reserve
static int get_crash_memory_ranges(struct memory_range **range, int *ranges) {
const char *iomem= proc_iomem();
- int memory_ranges = 0;
+ int memory_ranges = 0, gart = 0;
char line[MAX_LINE];
FILE *fp;
unsigned long long start, end;
+ uint64_t gart_start = 0, gart_end = 0;

fp = fopen(iomem, "r");
if (!fp) {
@@ -219,6 +220,11 @@ static int get_crash_memory_ranges(struc
type = RANGE_ACPI;
} else if(memcmp(str,"ACPI Non-volatile Storage\n",26) == 0 ) {
type = RANGE_ACPI_NVS;
+ } else if (memcmp(str, "GART\n", 5) == 0) {
+ gart_start = start;
+ gart_end = end;
+ gart = 1;
+ continue;
} else {
continue;
}
@@ -233,8 +239,14 @@ static int get_crash_memory_ranges(struc
memory_ranges++;
}
fclose(fp);
- if (exclude_crash_reserve_region(&memory_ranges) < 0)
+ if (exclude_region(&memory_ranges, crash_reserved_mem.start,
+ crash_reserved_mem.end) < 0)
return -1;
+ if (gart) {
+ /* exclude GART region if the system has one */
+ if (exclude_region(&memory_ranges, gart_start, gart_end) < 0)
+ return -1;
+ }
*range = crash_memory_range;
*ranges = memory_ranges;
#ifdef DEBUG
@@ -252,32 +264,27 @@ static int get_crash_memory_ranges(struc
/* Removes crash reserve region from list of memory chunks for whom elf
program
* headers have to be created. Assuming crash reserve region to be a single
* continuous area fully contained inside one of the memory chunks */
-static int exclude_crash_reserve_region(int *nr_ranges)
+static int exclude_region(int *nr_ranges, uint64_t start, uint64_t end)
{
int i, j, tidx = -1;
- unsigned long long cstart, cend;
struct memory_range temp_region;

- /* Crash reserved region. */
- cstart = crash_reserved_mem.start;
- cend = crash_reserved_mem.end;
-
for (i = 0; i < (*nr_ranges); i++) {
unsigned long long mstart, mend; mstart = crash_memory_range[i].start; mend = crash_memory_range[i].end;
- if (cstart < mend && cend > mstart) {
- if (cstart != mstart && cend != mend) {
+ if (start < mend && end > mstart) {
+ if (start != mstart && end != mend) {
/* Split memory region */
- crash_memory_range[i].end = cstart - 1;
- temp_region.start = cend + 1;
+ crash_memory_range[i].end = start - 1;
+ temp_region.start = end + 1;
temp_region.end = mend; temp_region.type = RANGE_RAM; tidx = i+1;
- } else if (cstart != mstart)
- crash_memory_range[i].end = cstart - 1;
+ } else if (start != mstart)
+ crash_memory_range[i].end = start - 1;
else
- crash_memory_range[i].start = cend + 1;
+ crash_memory_range[i].start = end + 1;
}
}
/* Insert split memory region, if any. */

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

author
#12 lore raw
In reply to Vivek Goyal
Quoted 8 lines · Vivek Goyal

Hi Chandru,

How much memory this system has got. Can you also paste the output of /proc/iomem of first kernel.

Does this system has GART? So looks like we are accessing some memory area which platform does not like. (We saw issues with GART in the past.)

The system has 8GB of RAM. /proc/iomem shows the following without mem=4G boot parameter

[root@abc]# cat /proc/iomem

00000000-0009afff : System RAM
0009b000-0009ffff : reserved
000e0000-000fffff : reserved
00100000-cff9f6ff : System RAM
00200000-0048adc9 : Kernel code
0048adca-005ee18f : Kernel data
0076d000-00823a4b : Kernel bss
02000000-11ffffff : Crash kernel
20000000-23ffffff : GART
cff9f700-cffa6fff : ACPI Tables
cffa7000-cfffffff : reserved
d4000000-d41fffff : PCI Bus 0000:01
d4000000-d41fffff : PCI Bus 0000:02
d4000000-d41fffff : PCI Bus 0000:03
d4000000-d41fffff : 0000:03:00.0
d4200000-d421ffff : 0000:00:05.0
d6000000-d6ffffff : PCI Bus 0000:0a
d7000000-d7ffffff : PCI Bus 0000:09
d8000000-d8ffffff : PCI Bus 0000:08
d9000000-d9ffffff : PCI Bus 0000:07
db000000-dcffffff : PCI Bus 0000:05
dc000000-dcffffff : PCI Bus 0000:06
de000000-e70fffff : PCI Bus 0000:01
de000000-e50fffff : PCI Bus 0000:02
de000000-e2ffffff : PCI Bus 0000:04
de000000-dfffffff : 0000:04:00.0
de000000-dfffffff : bnx2
e0000000-e1ffffff : 0000:04:00.1
e0000000-e1ffffff : bnx2
e4000000-e50fffff : PCI Bus 0000:03
e5000000-e500ffff : 0000:03:00.0
e5000000-e500ffff : mpt
e5010000-e5013fff : 0000:03:00.0
e5010000-e5013fff : mpt
e7000000-e701ffff : 0000:01:00.0
e8000000-efffffff : 0000:00:05.0
f3fed000-f3fedfff : 0000:00:0f.2
f3fed000-f3fedfff : ehci_hcd
f3fee000-f3feefff : 0000:00:0f.1
f3fee000-f3feefff : ohci_hcd
f3fef000-f3feffff : 0000:00:0f.0
f3fef000-f3feffff : ohci_hcd
f3ff0000-f3ffffff : 0000:00:05.0
f4000000-fbffffff : reserved
fa000000-faafffff : PCI MMCONFIG 0
fec00000-ffffffff : reserved
fec00000-fec00fff : IOAPIC 0
fec02000-fec02fff : IOAPIC 1
fed00000-fed003ff : HPET 0
fee00000-fee00fff : Local APIC
100000000-22fffffff : System RAM

With mem=4G /proc/iomem is as follows. The GART memory range seems to be missing here

00000000-0009afff : System RAM
0009b000-0009ffff : reserved
000e0000-000fffff : reserved
00100000-cff9f6ff : System RAM
00200000-0048adc9 : Kernel code
0048adca-005ee18f : Kernel data
0076d000-00823a4b : Kernel bss
02000000-11ffffff : Crash kernel
cff9f700-cffa6fff : ACPI Tables
cffa7000-cfffffff : reserved
d4000000-d41fffff : PCI Bus 0000:01
d4000000-d41fffff : PCI Bus 0000:02
d4000000-d41fffff : PCI Bus 0000:03
d4000000-d41fffff : 0000:03:00.0
d4200000-d421ffff : 0000:00:05.0
d6000000-d6ffffff : PCI Bus 0000:0a
d7000000-d7ffffff : PCI Bus 0000:09
d8000000-d8ffffff : PCI Bus 0000:08
d9000000-d9ffffff : PCI Bus 0000:07
db000000-dcffffff : PCI Bus 0000:05
dc000000-dcffffff : PCI Bus 0000:06
de000000-e70fffff : PCI Bus 0000:01
de000000-e50fffff : PCI Bus 0000:02
de000000-e2ffffff : PCI Bus 0000:04
de000000-dfffffff : 0000:04:00.0
de000000-dfffffff : bnx2
e0000000-e1ffffff : 0000:04:00.1
e0000000-e1ffffff : bnx2
e4000000-e50fffff : PCI Bus 0000:03
e5000000-e500ffff : 0000:03:00.0
e5000000-e500ffff : mpt
e5010000-e5013fff : 0000:03:00.0
e5010000-e5013fff : mpt
e7000000-e701ffff : 0000:01:00.0
e8000000-efffffff : 0000:00:05.0
f3fed000-f3fedfff : 0000:00:0f.2
f3fed000-f3fedfff : ehci_hcd
f3fee000-f3feefff : 0000:00:0f.1
f3fee000-f3feefff : ohci_hcd
f3fef000-f3feffff : 0000:00:0f.0
f3fef000-f3feffff : ohci_hcd
f3ff0000-f3ffffff : 0000:00:05.0
f4000000-fbffffff : reserved
fa000000-faafffff : PCI MMCONFIG 0
fec00000-ffffffff : reserved
fec00000-fec00fff : IOAPIC 0
fec02000-fec02fff : IOAPIC 1
fed00000-fed003ff : HPET 0
fee00000-fee00fff : Local APIC

Quoted 3 lines

Can you also provide /proc/vmcore ELF header (readelf output), in both the cases (mem=4G and without that).

ELF header with mem=4G

ELF Header:
Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
Class: ELF64
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: CORE (Core file)
Machine: Advanced Micro Devices X86-64
Version: 0x1
Entry point address: 0x0
Start of program headers: 64 (bytes into file) Start of section headers: 0 (bytes into file)
Flags: 0x0
Size of this header: 64 (bytes)
Size of program headers: 56 (bytes)
Number of program headers: 5
Size of section headers: 0 (bytes)
Number of section headers: 0
Section header string table index: 0

There are no sections in this file.

There are no sections in this file.

Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
NOTE 0x0000000000000158 0x0000000000000000 0x0000000000000000 0x0000000000000b20 0x0000000000000b20 0
LOAD 0x0000000000000c78 0xffffffff80200000 0x0000000000200000 0x0000000000624000 0x0000000000624000 RWE 0
LOAD 0x0000000000624c78 0xffff810000000000 0x0000000000000000 0x00000000000a0000 0x00000000000a0000 RWE 0
LOAD 0x00000000006c4c78 0xffff810000100000 0x0000000000100000 0x0000000001f00000 0x0000000001f00000 RWE 0
LOAD 0x00000000025c4c78 0xffff810012000000 0x0000000012000000 0x00000000bdf9f700 0x00000000bdf9f700 RWE 0

There is no dynamic section in this file.

There are no relocations in this file.

There are no unwind sections in this file.

No version information found in this file.

Notes at offset 0x00000158 with length 0x00000b20:
Owner Data size Description
CORE 0x00000150 NT_PRSTATUS (prstatus structure) CORE 0x00000150 NT_PRSTATUS (prstatus structure) CORE 0x00000150 NT_PRSTATUS (prstatus structure) CORE 0x00000150 NT_PRSTATUS (prstatus structure) CORE 0x00000150 NT_PRSTATUS (prstatus structure) CORE 0x00000150 NT_PRSTATUS (prstatus structure) CORE 0x00000150 NT_PRSTATUS (prstatus structure) CORE 0x00000150 NT_PRSTATUS (prstatus structure)

------------------------------------------------------------------------------------

ELF header without mem=4G

ELF Header:
Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
Class: ELF64
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: CORE (Core file)
Machine: Advanced Micro Devices X86-64
Version: 0x1
Entry point address: 0x0
Start of program headers: 64 (bytes into file) Start of section headers: 0 (bytes into file)
Flags: 0x0
Size of this header: 64 (bytes)
Size of program headers: 56 (bytes)
Number of program headers: 6
Size of section headers: 0 (bytes)
Number of section headers: 0
Section header string table index: 0

There are no sections in this file.

There are no sections in this file.

Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
NOTE 0x0000000000000190 0x0000000000000000 0x0000000000000000 0x0000000000000b20 0x0000000000000b20 0
LOAD 0x0000000000000cb0 0xffffffff80200000 0x0000000000200000 0x0000000000624000 0x0000000000624000 RWE 0
LOAD 0x0000000000624cb0 0xffff810000000000 0x0000000000000000 0x00000000000a0000 0x00000000000a0000 RWE 0
LOAD 0x00000000006c4cb0 0xffff810000100000 0x0000000000100000 0x0000000001f00000 0x0000000001f00000 RWE 0
LOAD 0x00000000025c4cb0 0xffff810012000000 0x0000000012000000 0x00000000bdf9f700 0x00000000bdf9f700 RWE 0
LOAD 0x00000000c05643b0 0xffff810100000000 0x0000000100000000 0x0000000130000000 0x0000000130000000 RWE 0

There is no dynamic section in this file.

There are no relocations in this file.

There are no unwind sections in this file.

No version information found in this file.

Notes at offset 0x00000190 with length 0x00000b20:
Owner Data size Description
CORE 0x00000150 NT_PRSTATUS (prstatus structure) CORE 0x00000150 NT_PRSTATUS (prstatus structure) CORE 0x00000150 NT_PRSTATUS (prstatus structure) CORE 0x00000150 NT_PRSTATUS (prstatus structure) CORE 0x00000150 NT_PRSTATUS (prstatus structure) CORE 0x00000150 NT_PRSTATUS (prstatus structure) CORE 0x00000150 NT_PRSTATUS (prstatus structure) CORE 0x00000150 NT_PRSTATUS (prstatus structure)

Quoted 7 lines

You can try putting some printk in /proc/vmcore code and see which physical memory area you are accessing when system goes bust. If in all the failure cases it is same physical memory area, then we can try to find what's so special about it.
Thanks
Vivek

The vmcore-incomplete files are of different sizes at different runs ( 18M, 32M.. ) and in case of n/w copy we get ( 190M, 198M ). I tried with the patch priovided by Bob Montgomery and it seems like it is working on this machine.

Thanks,
Chandru

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec