Skip to contentSkip to conversation

visible memory seems wrong in kexec crash dump kernel

DISCUSSION Chris Friesen 4 participants 14 messages Jul 11, 2013 – Jul 31, 2013

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi,

I'm running 2.6.34 with kexec 2.0.1 on a Freescale p5020-based system with 8GB of memory. (It's an embedded system and I can't do much about the fact that it's using older software.)

I booted the original kernel with "crashkernel=224M@32M" in the boot args. I then loaded the crash kernel with:

./kexec -p vmlinux.kontron.strip --append="root=/dev/mtdblock0 rw rootfstype=jffs2 panic=5 1 maxcpus=1 noirqdistrib reset_devices"

Then I triggered a crash:

echo 1 > /proc/sys/kernel/sysrq
echo c > /proc/sysrq-trigger

When the crash kernel came up, it had "elfcorehdr=48392K savemaxmem=8192M" in the kernel boot args, and "free" showed the system as having 7908652 KB of total memory instead of the expected 224MB.

Do I need to manually specify some options to kexec in order to preserve the memory of the original system? I seem to recall that kexec did this automatically on x86 systems.

Thanks,
Chris

#1 lore raw
Quoted 5 lines · Chris Friesen

Hi,

I'm running 2.6.34 with kexec 2.0.1 on a Freescale p5020-based system with 8GB of memory. (It's an embedded system and I can't do much about the fact that it's using older software.)

I should probably clarify this...I may be able to update kexec, I can't update the kernel but I can backport more recent code if necessary.

Looking at the version of kexec that I have, it seems like where x86 uses "memmap=" to specify the memory map usable by the capture kernel, powerpc does something different.

Chris

#2 lore raw
Quoted 7 lines · Chris Friesen
Quoted 5 lines · Chris Friesen

Hi,

I'm running 2.6.34 with kexec 2.0.1 on a Freescale p5020-based system with 8GB of memory. (It's an embedded system and I can't do much about the fact that it's using older software.)

Quoted 6 lines

I should probably clarify this...I may be able to update kexec, I can't update the kernel but I can backport more recent code if necessary.

Looking at the version of kexec that I have, it seems like where x86 uses "memmap=" to specify the memory map usable by the capture kernel, powerpc does something different.

After some experimenting, it looks like in the capture kernel /dev/oldmem might actually refer to the memory owned by the old kernel.

However, there doesn't seem to be anything preventing me from trampling it from within the capture kernel--I can create a tmpfs filesystem in memory and write gigs of data to it even though the capture kernel is only supposed to have 224MB.

I think I got it to work properly once, but since then I haven't been able to get uncorrupted data out of /dev/oldmem. (My original kernel reserves a chunk of memory for logging and passes the offset/size to the capture kernel via kexec kernel args.)

Chris

#3 lore raw
In reply to Chris Friesen
Quoted 11 lines · Chris Friesen
Quoted 5 lines · Chris Friesen

Hi,

I'm running 2.6.34 with kexec 2.0.1 on a Freescale p5020-based system with 8GB of memory. (It's an embedded system and I can't do much about the fact that it's using older software.)

I should probably clarify this...I may be able to update kexec, I can't update the kernel but I can backport more recent code if necessary.

Quoted 5 lines

Looking at the version of kexec that I have, it seems like where x86 uses "memmap=" to specify the memory map usable by the capture kernel, powerpc does something different.

From memory, it's been years, on powerpc we add properties to the memory

nodes in the device tree that specify which memory is usable. The properties are called "linux,usable-memory", and in modern kernels they are read in early_init_dt_scan_memory().

cheers

#4 lore raw
Quoted 12 lines · Michael Ellerman
Quoted 11 lines · Chris Friesen
Quoted 5 lines · Chris Friesen

Hi,

I'm running 2.6.34 with kexec 2.0.1 on a Freescale p5020-based system with 8GB of memory. (It's an embedded system and I can't do much about the fact that it's using older software.)

I should probably clarify this...I may be able to update kexec, I can't update the kernel but I can backport more recent code if necessary.

Quoted 8 lines
Quoted 3 lines

Looking at the version of kexec that I have, it seems like where x86 uses "memmap=" to specify the memory map usable by the capture kernel, powerpc does something different.

From memory, it's been years, on powerpc we add properties to the memory
nodes in the device tree that specify which memory is usable. The properties are called "linux,usable-memory", and in modern kernels they are read in early_init_dt_scan_memory().

I turned on the instrumentation in early_init_dt_scan_memory() and got the following when jumping to the capture kernel:

memory scan node memory, reg size 16, data: 0 0 2 0,
- 0 , 200000000

That 0x200000000 matches the fact that I'm seeing 8GB of memory available in the recovery kernel.

If I boot the original kernel with "crashkernel=224M@32M", should I expect that only 224MB is marked as "linux,usable-memory" in the recovery kernel?

Chris

#5 lore raw
Quoted 6 lines · Chris Friesen

I turned on the instrumentation in early_init_dt_scan_memory() and got the following when jumping to the capture kernel:

memory scan node memory, reg size 16, data: 0 0 2 0,
- 0 , 200000000

Quoted 6 lines

That 0x200000000 matches the fact that I'm seeing 8GB of memory available in the recovery kernel.

If I boot the original kernel with "crashkernel=224M@32M", should I expect that only 224MB is marked as "linux,usable-memory" in the recovery kernel?

I started looking at the kexec side of things, and I noticed something a bit odd. In most places dealing with the device tree in kexec it accepts either "memory" or "memory@" for the memory node name. In add_usable_mem_property() in arch/ppc64/fs2dt.c it seems to only accept "memory@".

Is this expected behaviour? It seems to be the same in current git versions of kexec-tools.

On my system I see "/proc/device-tree/memory".

If I modify add_usable_mem_property() to also accept "/memory" then my recovery kernel boots up with

physicalMemorySize = 0x10000000

which is 256MB (which is still a bit odd since I specified 224MB for the crashkernel).

However, it then hits the BUG() call at the end of mark_bootmem() in mm/bootmem.c.

Chris

#6 lore raw
Quoted 29 lines · Chris Friesen
Quoted 6 lines · Chris Friesen

I turned on the instrumentation in early_init_dt_scan_memory() and got the following when jumping to the capture kernel:

memory scan node memory, reg size 16, data: 0 0 2 0,
- 0 , 200000000

Quoted 6 lines

That 0x200000000 matches the fact that I'm seeing 8GB of memory available in the recovery kernel.

If I boot the original kernel with "crashkernel=224M@32M", should I expect that only 224MB is marked as "linux,usable-memory" in the recovery kernel?

I started looking at the kexec side of things, and I noticed something a bit odd. In most places dealing with the device tree in kexec it accepts either "memory" or "memory@" for the memory node name. In add_usable_mem_property() in arch/ppc64/fs2dt.c it seems to only accept "memory@".

Is this expected behaviour? It seems to be the same in current git versions of kexec-tools.

On my system I see "/proc/device-tree/memory".

If I modify add_usable_mem_property() to also accept "/memory" then my recovery kernel boots up with

Quoted 7 lines

physicalMemorySize = 0x10000000

which is 256MB (which is still a bit odd since I specified 224MB for the crashkernel).

However, it then hits the BUG() call at the end of mark_bootmem() in mm/bootmem.c.

One final thing and I'll stop replying to myself. :)

It looks like the problem is that some board-specific freescale code was calling lmb_reserve() with a base address in the 4GB range. It seems odd that lmb_reserve() didn't throw some kind of error when the recovery kernel was supposed to be limited to 224MB.

Rather than try and fix the bug, I turned off the (unneeded) config options related to the above lmb_reserve() calls and was able to successfully access the information I needed via /dev/oldmem.

The upshot is that there seems to be a number of things that could be improved:

1) kexec should accept "/memory" and not just "/memory@"
2) lmb_reserve() should really respect the crashkernel memory limit
3) the freescale stuff really shouldn't assume it can map things wherever it feels like

Chris

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#7 lore raw
Quoted 45 lines · Chris Friesen
Quoted 29 lines · Chris Friesen
Quoted 6 lines · Chris Friesen

I turned on the instrumentation in early_init_dt_scan_memory() and got the following when jumping to the capture kernel:

memory scan node memory, reg size 16, data: 0 0 2 0,
- 0 , 200000000

Quoted 6 lines

That 0x200000000 matches the fact that I'm seeing 8GB of memory available in the recovery kernel.

If I boot the original kernel with "crashkernel=224M@32M", should I expect that only 224MB is marked as "linux,usable-memory" in the recovery kernel?

I started looking at the kexec side of things, and I noticed something a bit odd. In most places dealing with the device tree in kexec it accepts either "memory" or "memory@" for the memory node name. In add_usable_mem_property() in arch/ppc64/fs2dt.c it seems to only accept "memory@".

Is this expected behaviour? It seems to be the same in current git versions of kexec-tools.

On my system I see "/proc/device-tree/memory".

If I modify add_usable_mem_property() to also accept "/memory" then my recovery kernel boots up with

Quoted 7 lines

physicalMemorySize = 0x10000000

which is 256MB (which is still a bit odd since I specified 224MB for the crashkernel).

However, it then hits the BUG() call at the end of mark_bootmem() in mm/bootmem.c.

One final thing and I'll stop replying to myself. :)

It looks like the problem is that some board-specific freescale code was calling lmb_reserve() with a base address in the 4GB range. It seems odd that lmb_reserve() didn't throw some kind of error when the recovery kernel was supposed to be limited to 224MB.

Quoted 8 lines

Rather than try and fix the bug, I turned off the (unneeded) config options related to the above lmb_reserve() calls and was able to successfully access the information I needed via /dev/oldmem.

The upshot is that there seems to be a number of things that could be improved:

1) kexec should accept "/memory" and not just "/memory@"

Yeah probably, I think folks tend to use "/memory@0" even if they only have a single memory node. But that does sound like a bug in kexec.

Quoted 1 line

2) lmb_reserve() should really respect the crashkernel memory limit

It's been replaced in mainline, so you'd have to check what the current code does in that situation.

Quoted 2 lines

3) the freescale stuff really shouldn't assume it can map things wherever it feels like

Agreed :)

cheers

#8 lore raw
Quoted 7 lines · Michael Ellerman
Quoted 6 lines
Quoted 6 lines

Is this expected behaviour? It seems to be the same in current git versions of kexec-tools.

On my system I see "/proc/device-tree/memory".

If I modify add_usable_mem_property() to also accept "/memory" then

my

This is a bug in your device-tree. The memory node should have a unit address which corresponds to it's reg property. I know people tend to skip it for "0" but it's bad practice.

So for memory starting at 0 it should be memory@0

Cheers,
Ben.

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#9 lore raw
Quoted 6 lines · Benjamin Herrenschmidt
Quoted 5 lines · Michael Ellerman
Quoted 5 lines
Quoted 5 lines

Is this expected behaviour? It seems to be the same in current git versions of kexec-tools.

On my system I see "/proc/device-tree/memory".

Quoted 8 lines
Quoted 2 lines
Quoted 1 line
Quoted 1 line

If I modify add_usable_mem_property() to also accept "/memory" then

my

This is a bug in your device-tree. The memory node should have a unit address which corresponds to it's reg property. I know people tend to skip it for "0" but it's bad practice.

So for memory starting at 0 it should be memory@0

There are a fair number of dts files in the kernel tree that don't specify an address for the memory node.

If the kernel accepts it without an address, it seems logical that kexec should as well.

Or maybe the kernel should just implicitly assume an address of zero and export it as such in /proc/device-tree?

Chris

#10 lore raw
Quoted 7 lines · Chris Friesen
Quoted 1 line

So for memory starting at 0 it should be memory@0

There are a fair number of dts files in the kernel tree that don't specify an address for the memory node.

If the kernel accepts it without an address, it seems logical that kexec should as well.

As long as kexec doesn't start being stupid when there are several nodes and doesn't pick up the "first one in device-tree order" instead of the one at 0...

I've been hit by that sort of bugs before (though not specifically in kexec).

Quoted 2 lines

Or maybe the kernel should just implicitly assume an address of zero and export it as such in /proc/device-tree?

I don't want /proc/device-tree to expose something different than what's in the actual device-tree, that would be the source for endless horrors.

We already are borderline with the occasional renaming we do in the case of duplicate name+property...

Cheers,
Ben.

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#11 lore raw
In reply to Chris Friesen
Quoted 54 lines · Chris Friesen
Quoted 34 lines · Chris Friesen

On 07/12/2013 03:08 PM, Chris Friesen wrote:
=20

Quoted 5 lines

I turned on the instrumentation in early_init_dt_scan_memory() and =20 got
the following when jumping to the capture kernel:
=20
memory scan node memory, reg size 16, data: 0 0 2 0,

Quoted 8 lines

- 0 , 200000000
=20
That 0x200000000 matches the fact that I'm seeing 8GB of memory available in the recovery kernel.
=20
If I boot the original kernel with "crashkernel=3D224M@32M", should I expect that only 224MB is marked as "linux,usable-memory" in the recovery kernel?

=20
I started looking at the kexec side of things, and I noticed =20 something a
bit odd. In most places dealing with the device tree in kexec it =20 accepts
either "memory" or "memory@" for the memory node name. In add_usable_mem_property() in arch/ppc64/fs2dt.c it seems to only =20 accept
"memory@".
=20
Is this expected behaviour? It seems to be the same in current git versions of kexec-tools.
=20
On my system I see "/proc/device-tree/memory".
=20
If I modify add_usable_mem_property() to also accept "/memory" then =20 my
recovery kernel boots up with
=20

Quoted 8 lines

physicalMemorySize =3D 0x10000000
=20
which is 256MB (which is still a bit odd since I specified 224MB for =20 the
crashkernel).
=20
However, it then hits the BUG() call at the end of mark_bootmem() in mm/bootmem.c.

=20
One final thing and I'll stop replying to myself. :)
=20
It looks like the problem is that some board-specific freescale code =20 was calling lmb_reserve() with a base address in the 4GB range. It =20 seems odd that lmb_reserve() didn't throw some kind of error when the =20 recovery kernel was supposed to be limited to 224MB.
=20
Rather than try and fix the bug, I turned off the (unneeded) config =20 options related to the above lmb_reserve() calls and was able to =20 successfully access the information I needed via /dev/oldmem.

Quoted 8 lines

=20
The upshot is that there seems to be a number of things that could be =20 improved:
=20
1) kexec should accept "/memory" and not just "/memory@"
2) lmb_reserve() should really respect the crashkernel memory limit
3) the freescale stuff really shouldn't assume it can map things =20 wherever it feels like

What "board-specific freescale code" are you referring to?

-Scott=

#12 lore raw

From: Scott Wood [[email protected]]
Sent: Monday, July 29, 2013 5:10 PM
To: Friesen, Christopher
Cc: Michael Ellerman; [email protected]; Paul Mackerras; [email protected]; Vivek Goyal
Subject: Re: visible memory seems wrong in kexec crash dump kernel

Quoted 7 lines · Chris Friesen

The upshot is that there seems to be a number of things that could be improved:

1) kexec should accept "/memory" and not just "/memory@"
2) lmb_reserve() should really respect the crashkernel memory limit
3) the freescale stuff really shouldn't assume it can map things wherever it feels like

What "board-specific freescale code" are you referring to?

-Scott

Sorry for the crappy quoting, I'm using a web outlook portal.

I've switched employers so I don't have access to the exact details any more. The system in question was a Kontron AM4150 which uses the P5020. As I recall, one of the Freescale drivers (I think it was the buffer or queue manager that the network driver makes use of) was attempting to call lmb_reserve() with a base address in the 4GB range even though the recovery kernel was limited to 224MB of memory.

While I've got your attention, the other thing that I found was that the "dpa" network driver didn't properly work in a kexec'd kernel even when given lots of memory. It would work for a little bit and then hang.

This was all on a Wind River Linux 4.3 system, which is based on 2.6.34. It's possible it's been fixed in more recent kernels. The testcase to see if current kernels are still affected should be fairly simple:

1) Take a P5020 board and boot it with NFS rootfs, reserving 224MB for crashkernel.
2) Load a crashkernel into it, configured to use NFS rootfs.
3) Trigger a panic on the main kernel.
4) Ensure it loads the recovery kernel properly, that it is limited to the correct amount of memory, that it has visibility of the original kernel's memory, that it has network connectivity and is stable when copying a few gigs of data over the network.

Chris

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#13 lore raw
Quoted 17 lines · Christopher

=20
From: Scott Wood [[email protected]]
Sent: Monday, July 29, 2013 5:10 PM
To: Friesen, Christopher
Cc: Michael Ellerman; [email protected]; Paul Mackerras; =20 [email protected]; Vivek Goyal
Subject: Re: visible memory seems wrong in kexec crash dump kernel =20

Quoted 1 line · Chris Friesen

The upshot is that there seems to be a number of things that could =20

be

Quoted 6 lines

improved:

1) kexec should accept "/memory" and not just "/memory@"
2) lmb_reserve() should really respect the crashkernel memory limit
3) the freescale stuff really shouldn't assume it can map things wherever it feels like

Quoted 14 lines

=20
What "board-specific freescale code" are you referring to? =20
-Scott
=20
=20
Sorry for the crappy quoting, I'm using a web outlook portal. =20
I've switched employers so I don't have access to the exact details =20 any more. The system in question was a Kontron AM4150 which uses the =20 P5020. As I recall, one of the Freescale drivers (I think it was the =20 buffer or queue manager that the network driver makes use of) was =20 attempting to call lmb_reserve() with a base address in the 4GB range =20 even though the recovery kernel was limited to 224MB of memory.

That's not "board specific" code, and it's not even mainline Linux =20 code. Unfortunately none of the datapath stuff is upstream, still.

Quoted 4 lines

While I've got your attention, the other thing that I found was that =20 the "dpa" network driver didn't properly work in a kexec'd kernel =20 even when given lots of memory. It would work for a little bit and =20 then hang.

I'm not particularly surprised by this. It doesn't help that there's =20 no way to do a device reset. :-(

Issues with Freescale SDK code should be reported on =20
https://community.freescale.com/, to [email protected], or to your =20 FAE.

-Scott=