Add +~800M crashkernel explaination
Subsystems (3)
PATCH 1/1 Add +~800M crashkernel explaination
When trying to configure crashkernel greater than about 800 MB, the kernel fails to allocate memory on x86 and x86_64. This is due to an undocumented limit that the crashkernel and other low memory items must be allocated below 896 MB unless the ",high" option is given. This updates the documentation to explain this and what I understand the limitations to be on the option.
--- Documentation/kdump/kdump.txt | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-)
Documentation/kdump/kdump.txt +17 -5
@@ -256,7 +256,9 @@ While the "crashkernel=size[@offset]" syntax is sufficient for most configurations, sometimes it's handy to have the reserved memory dependent on the value of System RAM -- that's mostly for distributors that pre-setup the kernel command line to avoid a unbootable system after some memory has -been removed from the machine. +been removed from the machine. If you need to allocate more than ~800M +for x86 or x86_64 then you must use the simple format as the format +',high' conflicts with the separators of ranges. The syntax is:
@@ -282,11 +284,21 @@ Boot into System Kernel 1) Update the boot loader (such as grub, yaboot, or lilo) configuration files as necessary. -2) Boot the system kernel with the boot parameter "crashkernel=Y@X", +2) Boot the system kernel with the boot parameter "crashkernel=Y[@X | ,high]", where Y specifies how much memory to reserve for the dump-capture kernel - and X specifies the beginning of this reserved memory. For example, - "crashkernel=64M@16M" tells the system kernel to reserve 64 MB of memory - starting at physical address 0x01000000 (16MB) for the dump-capture kernel. + and X specifies the beginning of this reserved memory or ',high' to load in + high memory. For example, "crashkernel=64M@16M" tells the system + kernel to reserve 64 MB of memory starting at physical address + 0x01000000 (16MB) for the dump-capture kernel. + + Specifying "crashkernel=1G,high" tells the system kernel to reserve 1 GB + of memory using high memory for the dump-capture kernel, there may also + be some low memory allocated as well. If you need more than ~800M for + the crash kernel to operate (volumes on FC/iSCSI, large volumes, systemd + added to the previous, etc), you need to specify ',high' since without + it crashkerenel has to try and fit under 896M along with some other + items and will fail to allocate memory. High memory may only be relevant + on x86 and x86_64. On x86 and x86_64, use "crashkernel=64M@16M".
Signature · 1 line
2.10.2
Quoted 6 lines · Robert LeBlanc
When trying to configure crashkernel greater than about 800 MB, the kernel fails to allocate memory on x86 and
x86_64. This is due to an undocumented limit that the crashkernel and other low memory items must be allocated below 896 MB unless the ",high" option is given. This updates the documentation to explain this and what I understand the limitations to be on the option.
This is true, but not very accurate. You found it's about 800M, it's becasue usually the current kernel need about 40M space to run, and some extra reservation before reserve_crashkernel invocation, another ~10M. However it's normal case, people may build modules into or have some special code to bloat kernel. This patch makes sense to address the low|high issue, it might be not good so determined to say ~800M.
Quoted 55 lines
Signed-off-by Robert LeBlanc <[email protected]>--- Documentation/kdump/kdump.txt | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-)Documentation/kdump/kdump.txt +17
-5@@ -256,7 +256,9 @@ While the "crashkernel=size[@offset]" syntax is sufficient for most configurations, sometimes it's handy to have the reserved memory dependent on the value of System RAM -- that's mostly for distributors that pre-setup the kernel command line to avoid a unbootable system after some memory has -been removed from the machine. +been removed from the machine. If you need to allocate more than ~800M +for x86 or x86_64 then you must use the simple format as the format +',high' conflicts with the separators of ranges. The syntax is:@@ -282,11 +284,21 @@ Boot into System Kernel 1) Update the boot loader (such as grub, yaboot, or lilo) configuration files as necessary. -2) Boot the system kernel with the boot parameter "crashkernel=Y@X", +2) Boot the system kernel with the boot parameter "crashkernel=Y[@X | ,high]", where Y specifies how much memory to reserve for the dump-capture kernel - and X specifies the beginning of this reserved memory. For example, - "crashkernel=64M@16M" tells the system kernel to reserve 64 MB of memory - starting at physical address 0x01000000 (16MB) for the dump-capture kernel. + and X specifies the beginning of this reserved memory or ',high' to load in + high memory. For example, "crashkernel=64M@16M" tells the system + kernel to reserve 64 MB of memory starting at physical address + 0x01000000 (16MB) for the dump-capture kernel. + + Specifying "crashkernel=1G,high" tells the system kernel to reserve 1 GB + of memory using high memory for the dump-capture kernel, there may also + be some low memory allocated as well. If you need more than ~800M for + the crash kernel to operate (volumes on FC/iSCSI, large volumes, systemd + added to the previous, etc), you need to specify ',high' since without + it crashkerenel has to try and fit under 896M along with some other + items and will fail to allocate memory. High memory may only be relevant + on x86 and x86_64. On x86 and x86_64, use "crashkernel=64M@16M".Signature · 7 lines
2.10.2 _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Quoted 8 lines · Baoquan He
Quoted 6 lines · Robert LeBlanc
When trying to configure crashkernel greater than about 800 MB, the kernel fails to allocate memory on x86 and
x86_64. This is due to an undocumented limit that the crashkernel and other low memory items must be allocated below 896 MB unless the ",high" option is given. This updates the documentation to explain this and what I understand the limitations to be on the option.
Quoted 6 lines
This is true, but not very accurate. You found it's about 800M, it's becasue usually the current kernel need about 40M space to run, and some extra reservation before
reserve_crashkernelinvocation, another ~10M. However it's normal case, people may build modules into or have some special code to bloat kernel. This patch makes sense to address the low|high issue, it might be not good so determined to say ~800M.
My testing showed that I could go anywhere from about 830M to 880M, depending on distro, kernel version, and stuff that you mentioned. I just thought some rule of thumb of when to consider using high would be good. People may not think that 800 MB is 'large' when you have 512 GB of RAM for instance. I thought about making 512 MB be the rule of thumb, but you can do a lot with ~300 MB.
I'm happy to adjust the wording, what would you recommend? Also, I'm not 100% sure that I got the cases covered correctly. I was surprised that I could not get it to work with the "new" format with the multiple ranges, and that specifying an offset would't work either, although the offset kind of makes sense. Do you know for sure that it doesn't work with ranges?
I tried,
crashkernel=256M-1G:128M,high,1G-4G:256M,high,4G-:512M,high
and
crashkernel=256M-1G:128M,1G-4G:256M,4G-:512M,high
and neither worked. It seems that a better separator would be ';' instead of ',' for ranges, then you could specify options better. Kind of hard to change now.
Quoted 48 lines
Quoted 48 lines
Signed-off-by Robert LeBlanc <[email protected]>--- Documentation/kdump/kdump.txt | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-)Documentation/kdump/kdump.txt +17
-5@@ -256,7 +256,9 @@ While the "crashkernel=size[@offset]" syntax is sufficient for most configurations, sometimes it's handy to have the reserved memory dependent on the value of System RAM -- that's mostly for distributors that pre-setup the kernel command line to avoid a unbootable system after some memory has -been removed from the machine. +been removed from the machine. If you need to allocate more than ~800M +for x86 or x86_64 then you must use the simple format as the format +',high' conflicts with the separators of ranges. The syntax is:@@ -282,11 +284,21 @@ Boot into System Kernel 1) Update the boot loader (such as grub, yaboot, or lilo) configuration files as necessary. -2) Boot the system kernel with the boot parameter "crashkernel=Y@X", +2) Boot the system kernel with the boot parameter "crashkernel=Y[@X | ,high]", where Y specifies how much memory to reserve for the dump-capture kernel - and X specifies the beginning of this reserved memory. For example, - "crashkernel=64M@16M" tells the system kernel to reserve 64 MB of memory - starting at physical address 0x01000000 (16MB) for the dump-capture kernel. + and X specifies the beginning of this reserved memory or ',high' to load in + high memory. For example, "crashkernel=64M@16M" tells the system + kernel to reserve 64 MB of memory starting at physical address + 0x01000000 (16MB) for the dump-capture kernel. + + Specifying "crashkernel=1G,high" tells the system kernel to reserve 1 GB + of memory using high memory for the dump-capture kernel, there may also + be some low memory allocated as well. If you need more than ~800M for + the crash kernel to operate (volumes on FC/iSCSI, large volumes, systemd + added to the previous, etc), you need to specify ',high' since without + it crashkerenel has to try and fit under 896M along with some other + items and will fail to allocate memory. High memory may only be relevant + on x86 and x86_64. On x86 and x86_64, use "crashkernel=64M@16M".
Quoted 8 lines
Quoted 8 lines
--
2.10.2_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
Quoted 14 lines · Robert LeBlanc
Quoted 7 lines · Baoquan He
Quoted 6 lines · Robert LeBlanc
When trying to configure crashkernel greater than about 800 MB, the kernel fails to allocate memory on x86 and
x86_64. This is due to an undocumented limit that the crashkernel and other low memory items must be allocated below 896 MB unless the ",high" option is given. This updates the documentation to explain this and what I understand the limitations to be on the option.Quoted 6 lines
This is true, but not very accurate. You found it's about 800M, it's becasue usually the current kernel need about 40M space to run, and some extra reservation before
reserve_crashkernelinvocation, another ~10M. However it's normal case, people may build modules into or have some special code to bloat kernel. This patch makes sense to address the low|high issue, it might be not good so determined to say ~800M.
Quoted 6 lines
My testing showed that I could go anywhere from about 830M to 880M, depending on distro, kernel version, and stuff that you mentioned. I just thought some rule of thumb of when to consider using high would be good. People may not think that 800 MB is 'large' when you have 512 GB of RAM for instance. I thought about making 512 MB be the rule of thumb, but you can do a lot with ~300 MB.
Hi Robert,
I think you are correct.
For x86, the kernel uses memblock to locate the proper range starts from 16MB to some "end", without "high" prefix, "end" is CRASH_ADDR_LOW_MAX, otherwise CRASH_ADDR_HIGH_MAX.
You can find the definition for both 32-bit and 64-bit:
#ifdef CONFIG_X86_32
# define CRASH_ADDR_LOW_MAX (512 << 20) # define CRASH_ADDR_HIGH_MAX (512 << 20) #else # define CRASH_ADDR_LOW_MAX (896UL << 20) # define CRASH_ADDR_HIGH_MAX MAXMEM #endif
as some memory was already allocated by the kernel, which means it's highly likely to get a reservation failure after specifying a crashkernel value near 800MB(for x86_64) which was what you met. But we can't get the exact threshold, but it would be better if there is some explanation accordingly in the document.
Quoted 12 lines
I'm happy to adjust the wording, what would you recommend? Also, I'm not 100% sure that I got the cases covered correctly. I was surprised that I could not get it to work with the "new" format with the multiple ranges, and that specifying an offset would't work either, although the offset kind of makes sense. Do you know for sure that it doesn't work with ranges?
I tried,
crashkernel=256M-1G:128M,high,1G-4G:256M,high,4G-:512M,high
Quoted 7 lines
and
crashkernel=256M-1G:128M,1G-4G:256M,4G-:512M,high
and neither worked. It seems that a better separator would be ';' instead of ',' for ranges, then you could specify options better. Kind of hard to change now.
For "crashkernel=range1:size1[,range2:size2,...][@offset]"
I'm afraid it doesn't support "high" prefix in the current implementation, so there is no guarantee. I guess we can drop a note to eliminate the confusion.
Regards,
Xunlei
Quoted 62 lines
Quoted 46 lines
Quoted 46 lines
Signed-off-by Robert LeBlanc <[email protected]>--- Documentation/kdump/kdump.txt | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-)Documentation/kdump/kdump.txt +17
-5@@ -256,7 +256,9 @@ While the "crashkernel=size[@offset]" syntax is sufficient for most configurations, sometimes it's handy to have the reserved memory dependent on the value of System RAM -- that's mostly for distributors that pre-setup the kernel command line to avoid a unbootable system after some memory has -been removed from the machine. +been removed from the machine. If you need to allocate more than ~800M +for x86 or x86_64 then you must use the simple format as the format +',high' conflicts with the separators of ranges. The syntax is:@@ -282,11 +284,21 @@ Boot into System Kernel 1) Update the boot loader (such as grub, yaboot, or lilo) configuration files as necessary. -2) Boot the system kernel with the boot parameter "crashkernel=Y@X", +2) Boot the system kernel with the boot parameter "crashkernel=Y[@X | ,high]", where Y specifies how much memory to reserve for the dump-capture kernel - and X specifies the beginning of this reserved memory. For example, - "crashkernel=64M@16M" tells the system kernel to reserve 64 MB of memory - starting at physical address 0x01000000 (16MB) for the dump-capture kernel. + and X specifies the beginning of this reserved memory or ',high' to load in + high memory. For example, "crashkernel=64M@16M" tells the system + kernel to reserve 64 MB of memory starting at physical address + 0x01000000 (16MB) for the dump-capture kernel. + + Specifying "crashkernel=1G,high" tells the system kernel to reserve 1 GB + of memory using high memory for the dump-capture kernel, there may also + be some low memory allocated as well. If you need more than ~800M for + the crash kernel to operate (volumes on FC/iSCSI, large volumes, systemd + added to the previous, etc), you need to specify ',high' since without + it crashkerenel has to try and fit under 896M along with some other + items and will fail to allocate memory. High memory may only be relevant + on x86 and x86_64. On x86 and x86_64, use "crashkernel=64M@16M".Quoted 8 lines
Quoted 8 lines
--
2.10.2_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 37 lines · Xunlei Pang
Quoted 14 lines · Robert LeBlanc
Quoted 7 lines · Baoquan He
Quoted 6 lines · Robert LeBlanc
When trying to configure crashkernel greater than about 800 MB, the kernel fails to allocate memory on x86 and
x86_64. This is due to an undocumented limit that the crashkernel and other low memory items must be allocated below 896 MB unless the ",high" option is given. This updates the documentation to explain this and what I understand the limitations to be on the option.Quoted 6 lines
This is true, but not very accurate. You found it's about 800M, it's becasue usually the current kernel need about 40M space to run, and some extra reservation before
reserve_crashkernelinvocation, another ~10M. However it's normal case, people may build modules into or have some special code to bloat kernel. This patch makes sense to address the low|high issue, it might be not good so determined to say ~800M.Quoted 6 lines
My testing showed that I could go anywhere from about 830M to 880M, depending on distro, kernel version, and stuff that you mentioned. I just thought some rule of thumb of when to consider using high would be good. People may not think that 800 MB is 'large' when you have 512 GB of RAM for instance. I thought about making 512 MB be the rule of thumb, but you can do a lot with ~300 MB.
Hi Robert,
I think you are correct.
For x86, the kernel uses memblock to locate the proper range starts from 16MB to some "end", without "high" prefix, "end" is
CRASH_ADDR_LOW_MAX, otherwiseCRASH_ADDR_HIGH_MAX.You can find the definition for both 32-bit and 64-bit:
#ifdefCONFIG_X86_32# define CRASH_ADDR_LOW_MAX (512 << 20) # define CRASH_ADDR_HIGH_MAX (512 << 20) #else # define CRASH_ADDR_LOW_MAX (896UL << 20) # define CRASH_ADDR_HIGH_MAX MAXMEM #endif
Quoted 3 lines
as some memory was already allocated by the kernel, which means it's highly likely to get a reservation failure after specifying a crashkernel value near 800MB(for
x86_64) which was what you met. But we can't get the exact threshold, but it would be better if there is some explanation accordingly in the document.
But there is another point:
If you specify the base using crashkernel=size[KMG][@offset[KMG]], for example "crashkernel=1024M@0x10000000", there is no such limitation, and you may get a successful reservation. I have no idea why the design is so different.
Regards,
Xunlei
Quoted 93 lines
Quoted 12 lines
I'm happy to adjust the wording, what would you recommend? Also, I'm not 100% sure that I got the cases covered correctly. I was surprised that I could not get it to work with the "new" format with the multiple ranges, and that specifying an offset would't work either, although the offset kind of makes sense. Do you know for sure that it doesn't work with ranges?
I tried,
crashkernel=256M-1G:128M,high,1G-4G:256M,high,4G-:512M,high
Quoted 7 lines
and
crashkernel=256M-1G:128M,1G-4G:256M,4G-:512M,high
and neither worked. It seems that a better separator would be ';' instead of ',' for ranges, then you could specify options better. Kind of hard to change now.
For "crashkernel=range1:size1[,range2:size2,...][@offset]"
I'm afraid it doesn't support "high" prefix in the current implementation, so there is no guarantee. I guess we can drop a note to eliminate the confusion.Regards,
XunleiQuoted 58 lines
Quoted 46 lines
Quoted 46 lines
Signed-off-by Robert LeBlanc <[email protected]>--- Documentation/kdump/kdump.txt | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-)Documentation/kdump/kdump.txt +17
-5@@ -256,7 +256,9 @@ While the "crashkernel=size[@offset]" syntax is sufficient for most configurations, sometimes it's handy to have the reserved memory dependent on the value of System RAM -- that's mostly for distributors that pre-setup the kernel command line to avoid a unbootable system after some memory has -been removed from the machine. +been removed from the machine. If you need to allocate more than ~800M +for x86 or x86_64 then you must use the simple format as the format +',high' conflicts with the separators of ranges. The syntax is:@@ -282,11 +284,21 @@ Boot into System Kernel 1) Update the boot loader (such as grub, yaboot, or lilo) configuration files as necessary. -2) Boot the system kernel with the boot parameter "crashkernel=Y@X", +2) Boot the system kernel with the boot parameter "crashkernel=Y[@X | ,high]", where Y specifies how much memory to reserve for the dump-capture kernel - and X specifies the beginning of this reserved memory. For example, - "crashkernel=64M@16M" tells the system kernel to reserve 64 MB of memory - starting at physical address 0x01000000 (16MB) for the dump-capture kernel. + and X specifies the beginning of this reserved memory or ',high' to load in + high memory. For example, "crashkernel=64M@16M" tells the system + kernel to reserve 64 MB of memory starting at physical address + 0x01000000 (16MB) for the dump-capture kernel. + + Specifying "crashkernel=1G,high" tells the system kernel to reserve 1 GB + of memory using high memory for the dump-capture kernel, there may also + be some low memory allocated as well. If you need more than ~800M for + the crash kernel to operate (volumes on FC/iSCSI, large volumes, systemd + added to the previous, etc), you need to specify ',high' since without + it crashkerenel has to try and fit under 896M along with some other + items and will fail to allocate memory. High memory may only be relevant + on x86 and x86_64. On x86 and x86_64, use "crashkernel=64M@16M".Quoted 8 lines
Quoted 8 lines
--
2.10.2_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1Quoted 4 lines
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 38 lines · Xunlei Pang
Quoted 14 lines · Robert LeBlanc
Quoted 7 lines · Baoquan He
Quoted 6 lines · Robert LeBlanc
When trying to configure crashkernel greater than about 800 MB, the kernel fails to allocate memory on x86 and
x86_64. This is due to an undocumented limit that the crashkernel and other low memory items must be allocated below 896 MB unless the ",high" option is given. This updates the documentation to explain this and what I understand the limitations to be on the option.Quoted 6 lines
This is true, but not very accurate. You found it's about 800M, it's becasue usually the current kernel need about 40M space to run, and some extra reservation before
reserve_crashkernelinvocation, another ~10M. However it's normal case, people may build modules into or have some special code to bloat kernel. This patch makes sense to address the low|high issue, it might be not good so determined to say ~800M.Quoted 6 lines
My testing showed that I could go anywhere from about 830M to 880M, depending on distro, kernel version, and stuff that you mentioned. I just thought some rule of thumb of when to consider using high would be good. People may not think that 800 MB is 'large' when you have 512 GB of RAM for instance. I thought about making 512 MB be the rule of thumb, but you can do a lot with ~300 MB.
Hi Robert,
I think you are correct.
For x86, the kernel uses memblock to locate the proper range starts from 16MB to some "end", without "high" prefix, "end" is
CRASH_ADDR_LOW_MAX, otherwiseCRASH_ADDR_HIGH_MAX.You can find the definition for both 32-bit and 64-bit:
#ifdefCONFIG_X86_32# define CRASH_ADDR_LOW_MAX (512 << 20) # define CRASH_ADDR_HIGH_MAX (512 << 20) #else # define CRASH_ADDR_LOW_MAX (896UL << 20) # define CRASH_ADDR_HIGH_MAX MAXMEM #endif
Quoted 3 lines
as some memory was already allocated by the kernel, which means it's highly likely to get a reservation failure after specifying a crashkernel value near 800MB(for
x86_64) which was what you met. But we can't get the exact threshold, but it would be better if there is some explanation accordingly in the document.
To make sure I'm understanding what you are say, you want me to go into a bit more detail about the limitation and specify the differences between x86 and x86_64, right?
Quoted 15 lines
Quoted 15 lines
I'm happy to adjust the wording, what would you recommend? Also, I'm not 100% sure that I got the cases covered correctly. I was surprised that I could not get it to work with the "new" format with the multiple ranges, and that specifying an offset would't work either, although the offset kind of makes sense. Do you know for sure that it doesn't work with ranges?
I tried,
crashkernel=256M-1G:128M,high,1G-4G:256M,high,4G-:512M,high
and
crashkernel=256M-1G:128M,1G-4G:256M,4G-:512M,high
Quoted 7 lines
Quoted 3 lines
and neither worked. It seems that a better separator would be ';' instead of ',' for ranges, then you could specify options better. Kind of hard to change now.
For "crashkernel=range1:size1[,range2:size2,...][@offset]"
I'm afraid it doesn't support "high" prefix in the current implementation, so there is no guarantee. I guess we can drop a note to eliminate the confusion.
I tried to express in the extended syntax section that ',high' is not available and you have to use the 'simple' format. Do you think this needs to be expanded as well?
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
Quoted 63 lines
Quoted 62 lines
Quoted 46 lines
Quoted 46 lines
Signed-off-by Robert LeBlanc <[email protected]>--- Documentation/kdump/kdump.txt | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-)Documentation/kdump/kdump.txt +17
-5@@ -256,7 +256,9 @@ While the "crashkernel=size[@offset]" syntax is sufficient for most configurations, sometimes it's handy to have the reserved memory dependent on the value of System RAM -- that's mostly for distributors that pre-setup the kernel command line to avoid a unbootable system after some memory has -been removed from the machine. +been removed from the machine. If you need to allocate more than ~800M +for x86 or x86_64 then you must use the simple format as the format +',high' conflicts with the separators of ranges. The syntax is:@@ -282,11 +284,21 @@ Boot into System Kernel 1) Update the boot loader (such as grub, yaboot, or lilo) configuration files as necessary. -2) Boot the system kernel with the boot parameter "crashkernel=Y@X", +2) Boot the system kernel with the boot parameter "crashkernel=Y[@X | ,high]", where Y specifies how much memory to reserve for the dump-capture kernel - and X specifies the beginning of this reserved memory. For example, - "crashkernel=64M@16M" tells the system kernel to reserve 64 MB of memory - starting at physical address 0x01000000 (16MB) for the dump-capture kernel. + and X specifies the beginning of this reserved memory or ',high' to load in + high memory. For example, "crashkernel=64M@16M" tells the system + kernel to reserve 64 MB of memory starting at physical address + 0x01000000 (16MB) for the dump-capture kernel. + + Specifying "crashkernel=1G,high" tells the system kernel to reserve 1 GB + of memory using high memory for the dump-capture kernel, there may also + be some low memory allocated as well. If you need more than ~800M for + the crash kernel to operate (volumes on FC/iSCSI, large volumes, systemd + added to the previous, etc), you need to specify ',high' since without + it crashkerenel has to try and fit under 896M along with some other + items and will fail to allocate memory. High memory may only be relevant + on x86 and x86_64. On x86 and x86_64, use "crashkernel=64M@16M".Quoted 8 lines
Quoted 8 lines
--
2.10.2_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 38 lines · Robert LeBlanc
Quoted 37 lines · Xunlei Pang
Quoted 14 lines · Robert LeBlanc
Quoted 7 lines · Baoquan He
Quoted 6 lines · Robert LeBlanc
When trying to configure crashkernel greater than about 800 MB, the kernel fails to allocate memory on x86 and
x86_64. This is due to an undocumented limit that the crashkernel and other low memory items must be allocated below 896 MB unless the ",high" option is given. This updates the documentation to explain this and what I understand the limitations to be on the option.Quoted 6 lines
This is true, but not very accurate. You found it's about 800M, it's becasue usually the current kernel need about 40M space to run, and some extra reservation before
reserve_crashkernelinvocation, another ~10M. However it's normal case, people may build modules into or have some special code to bloat kernel. This patch makes sense to address the low|high issue, it might be not good so determined to say ~800M.Quoted 6 lines
My testing showed that I could go anywhere from about 830M to 880M, depending on distro, kernel version, and stuff that you mentioned. I just thought some rule of thumb of when to consider using high would be good. People may not think that 800 MB is 'large' when you have 512 GB of RAM for instance. I thought about making 512 MB be the rule of thumb, but you can do a lot with ~300 MB.
Hi Robert,
I think you are correct.
For x86, the kernel uses memblock to locate the proper range starts from 16MB to some "end", without "high" prefix, "end" is
CRASH_ADDR_LOW_MAX, otherwiseCRASH_ADDR_HIGH_MAX.You can find the definition for both 32-bit and 64-bit:
#ifdefCONFIG_X86_32# define CRASH_ADDR_LOW_MAX (512 << 20) # define CRASH_ADDR_HIGH_MAX (512 << 20) #else # define CRASH_ADDR_LOW_MAX (896UL << 20) # define CRASH_ADDR_HIGH_MAX MAXMEM #endif
Quoted 6 lines
Quoted 3 lines
as some memory was already allocated by the kernel, which means it's highly likely to get a reservation failure after specifying a crashkernel value near 800MB(for
x86_64) which was what you met. But we can't get the exact threshold, but it would be better if there is some explanation accordingly in the document.To make sure I'm understanding what you are say, you want me to go into a bit more detail about the limitation and specify the differences between x86 and
x86_64, right?
Yeah, it would be better to have one, at least to mention the different upper bounds.
As I replied in another post, if you really want to detail the behaviour, should mention "crashkernel=size[KMG][@offset[KMG]]" with @offset[KMG] specified explicitly, after all, it's handled differently with no upper bound limitation, but doing this may put the first kernel at the risk of lacking low memory(some devices require 32bit DMA), must use it with care because the kernel will assume users are aware of what they are doing and make a successful reservation as long as the given range is available.
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
I'm happy to adjust the wording, what would you recommend? Also, I'm not 100% sure that I got the cases covered correctly. I was surprised that I could not get it to work with the "new" format with the multiple ranges, and that specifying an offset would't work either, although the offset kind of makes sense. Do you know for sure that it doesn't work with ranges?
I tried,
crashkernel=256M-1G:128M,high,1G-4G:256M,high,4G-:512M,high
and
crashkernel=256M-1G:128M,1G-4G:256M,4G-:512M,high
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
and neither worked. It seems that a better separator would be ';' instead of ',' for ranges, then you could specify options better. Kind of hard to change now.
For "crashkernel=range1:size1[,range2:size2,...][@offset]"
I'm afraid it doesn't support "high" prefix in the current implementation, so there is no guarantee. I guess we can drop a note to eliminate the confusion.I tried to express in the extended syntax section that ',high' is not available and you have to use the 'simple' format. Do you think this
ditto
Quoted 1 line
needs to be expanded as well?
If you really have good reasons or use cases, please try it :-)
Regards,
Xunlei
Quoted 72 lines
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1Quoted 58 lines
Quoted 58 lines
Quoted 46 lines
Quoted 46 lines
Signed-off-by Robert LeBlanc <[email protected]>--- Documentation/kdump/kdump.txt | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-)Documentation/kdump/kdump.txt +17
-5@@ -256,7 +256,9 @@ While the "crashkernel=size[@offset]" syntax is sufficient for most configurations, sometimes it's handy to have the reserved memory dependent on the value of System RAM -- that's mostly for distributors that pre-setup the kernel command line to avoid a unbootable system after some memory has -been removed from the machine. +been removed from the machine. If you need to allocate more than ~800M +for x86 or x86_64 then you must use the simple format as the format +',high' conflicts with the separators of ranges. The syntax is:@@ -282,11 +284,21 @@ Boot into System Kernel 1) Update the boot loader (such as grub, yaboot, or lilo) configuration files as necessary. -2) Boot the system kernel with the boot parameter "crashkernel=Y@X", +2) Boot the system kernel with the boot parameter "crashkernel=Y[@X | ,high]", where Y specifies how much memory to reserve for the dump-capture kernel - and X specifies the beginning of this reserved memory. For example, - "crashkernel=64M@16M" tells the system kernel to reserve 64 MB of memory - starting at physical address 0x01000000 (16MB) for the dump-capture kernel. + and X specifies the beginning of this reserved memory or ',high' to load in + high memory. For example, "crashkernel=64M@16M" tells the system + kernel to reserve 64 MB of memory starting at physical address + 0x01000000 (16MB) for the dump-capture kernel. + + Specifying "crashkernel=1G,high" tells the system kernel to reserve 1 GB + of memory using high memory for the dump-capture kernel, there may also + be some low memory allocated as well. If you need more than ~800M for + the crash kernel to operate (volumes on FC/iSCSI, large volumes, systemd + added to the previous, etc), you need to specify ',high' since without + it crashkerenel has to try and fit under 896M along with some other + items and will fail to allocate memory. High memory may only be relevant + on x86 and x86_64. On x86 and x86_64, use "crashkernel=64M@16M".Quoted 8 lines
Quoted 8 lines
--
2.10.2_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1Quoted 4 lines
Quoted 4 lines
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 6 lines · Xunlei Pang
As I replied in another post, if you really want to detail the behaviour, should mention "crashkernel=size[KMG][@offset[KMG]]" with @offset[KMG] specified explicitly, after all, it's handled differently with no upper bound limitation, but doing this may put the first kernel at the risk of lacking low memory(some devices require 32bit DMA), must use it with care because the kernel will assume users are aware of what they are doing and make a successful reservation as long as the given range is available.
crashkernel=1024M@0x10000000
I can't get the offset to work. It seems that it allocates the space and loads the crash kernel, but I couldn't get it to actually boot into the crash kernel. Does it work for you? I'm using the 4.9 kernel.
Thanks
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
Quoted 7 lines · Robert LeBlanc
Quoted 6 lines · Xunlei Pang
As I replied in another post, if you really want to detail the behaviour, should mention "crashkernel=size[KMG][@offset[KMG]]" with @offset[KMG] specified explicitly, after all, it's handled differently with no upper bound limitation, but doing this may put the first kernel at the risk of lacking low memory(some devices require 32bit DMA), must use it with care because the kernel will assume users are aware of what they are doing and make a successful reservation as long as the given range is available.
Quoted 5 lines
crashkernel=1024M@0x10000000
I can't get the offset to work. It seems that it allocates the space and loads the crash kernel, but I couldn't get it to actually boot into the crash kernel. Does it work for you? I'm using the 4.9 kernel.
Not sure what is the problem you met, but kdump kernel boots well using 4.9 on my x86_64 machine.
Regards,
Xunlei
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec