Skip to contentSkip to conversation

x86, kdump: crashkernel=X try to reserve below 896M first, then try below 4G, then MAXMEM

PATCH In review WANG Chao 4 participants 26 messages Oct 14, 2013 – Nov 4, 2013

Subsystems (3)
lore
Share & tools
lore

WANG Chao

author
#0 lore raw

PATCH 1/1 x86, kdump: crashkernel=X try to reserve below 896M first, then try below 4G, then MAXMEM

Now crashkernel=X will fail out if there's not enough memory at low (below 896M). What makes sense for crashkernel=X would be:

- First try to reserve X below 896M (for being compatible with old kexec-tools).
- If fails, try to reserve X below 4G (swiotlb need to stay below 4G).
- If fails, try to reserve X from MAXMEM top down.

So that user can easily reserve large memory with crashkernel=X instead of crashkernel=X,high. It's more transparent and user-friendly.

If crashkernel is large and the reserved is beyond 896M, old kexec-tools won't be compatible with new kernel for most of time.

kexec will fail out immediately in this case. But the failure could be expected, because old kexec users should not try to reserve that large amount of memory at the first place.

On the other hand, old kexec also will fail on old kernel when there's not enough low memory to reserve a large crash kernel area. So the failure of old kexec is consistent between old kernel and new kernel.

---
 arch/x86/kernel/setup.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
arch/x86/kernel/setup.c +14 -0
index f0de629..38e6c1f 100644
@@ -593,6 +593,20 @@ static void __init reserve_crashkernel(void)
 					high ? CRASH_KERNEL_ADDR_HIGH_MAX :
 					       CRASH_KERNEL_ADDR_LOW_MAX,
 					crash_size, alignment);
+		/*
+		 * crashkernel=X reserve below 896M fails? Try below 4G
+		 */
+		if (!high && !crash_base)
+			crash_base = memblock_find_in_range(alignment,
+						(1ULL << 32),
+						crash_size, alignment);
+		/*
+		 * crashkernel=X reserve below 4G fails? Try MAXMEM
+		 */
+		if (!high && !crash_base)
+			crash_base = memblock_find_in_range(alignment,
+						CRASH_KERNEL_ADDR_HIGH_MAX,
+						crash_size, alignment);
 
 		if (!crash_base) {
 			pr_info("crashkernel reservation failed - No suitable area found.\n");
Signature · 7 lines
1.8.3.1


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#1 lore raw
Quoted 43 lines · WANG Chao

Now crashkernel=X will fail out if there's not enough memory at low (below 896M). What makes sense for crashkernel=X would be:

- First try to reserve X below 896M (for being compatible with old kexec-tools).
- If fails, try to reserve X below 4G (swiotlb need to stay below 4G).
- If fails, try to reserve X from MAXMEM top down.

So that user can easily reserve large memory with crashkernel=X instead of crashkernel=X,high. It's more transparent and user-friendly.

If crashkernel is large and the reserved is beyond 896M, old kexec-tools won't be compatible with new kernel for most of time.

kexec will fail out immediately in this case. But the failure could be expected, because old kexec users should not try to reserve that large amount of memory at the first place.

On the other hand, old kexec also will fail on old kernel when there's not enough low memory to reserve a large crash kernel area. So the failure of old kexec is consistent between old kernel and new kernel.

---
 arch/x86/kernel/setup.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
arch/x86/kernel/setup.c +8 -0
index f0de629..38e6c1f 100644
@@ -593,6 +593,20 @@ static void __init reserve_crashkernel(void)
                                        high ? CRASH_KERNEL_ADDR_HIGH_MAX :
                                               CRASH_KERNEL_ADDR_LOW_MAX,
                                        crash_size, alignment);
+               /*
+                * crashkernel=X reserve below 896M fails? Try below 4G
+                */
+               if (!high && !crash_base)
+                       crash_base = memblock_find_in_range(alignment,
+                                               (1ULL << 32),
+                                               crash_size, alignment);
+               /*

arch/x86/kernel/setup.c static void __init reserve_crashkernel(void)

Quoted 10 lines
arch/x86/kernel/setup.c +6 -1
+                * crashkernel=X reserve below 4G fails? Try MAXMEM
+                */
+               if (!high && !crash_base)
+                       crash_base = memblock_find_in_range(alignment,
+                                               CRASH_KERNEL_ADDR_HIGH_MAX,
+                                               crash_size, alignment);
 
                if (!crash_base) {
                        pr_info("crashkernel reservation failed - No suitable area found.\n");
--

User should change crashkernel=X to crashkernel=X,high.

As user could forget to update kexec-tools to utilize ",high" feature, and get kdump later fail later.

Yinghai

author
#2 lore raw
Quoted 51 lines · Yinghai Lu
Quoted 50 lines · WANG Chao

Now crashkernel=X will fail out if there's not enough memory at low (below 896M). What makes sense for crashkernel=X would be:

- First try to reserve X below 896M (for being compatible with old kexec-tools).
- If fails, try to reserve X below 4G (swiotlb need to stay below 4G).
- If fails, try to reserve X from MAXMEM top down.

So that user can easily reserve large memory with crashkernel=X instead of crashkernel=X,high. It's more transparent and user-friendly.

If crashkernel is large and the reserved is beyond 896M, old kexec-tools won't be compatible with new kernel for most of time.

kexec will fail out immediately in this case. But the failure could be expected, because old kexec users should not try to reserve that large amount of memory at the first place.

On the other hand, old kexec also will fail on old kernel when there's not enough low memory to reserve a large crash kernel area. So the failure of old kexec is consistent between old kernel and new kernel.

---
 arch/x86/kernel/setup.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
arch/x86/kernel/setup.c +14 -0
index f0de629..38e6c1f 100644
@@ -593,6 +593,20 @@ static void __init reserve_crashkernel(void)
                                        high ? CRASH_KERNEL_ADDR_HIGH_MAX :
                                               CRASH_KERNEL_ADDR_LOW_MAX,
                                        crash_size, alignment);
+               /*
+                * crashkernel=X reserve below 896M fails? Try below 4G
+                */
+               if (!high && !crash_base)
+                       crash_base = memblock_find_in_range(alignment,
+                                               (1ULL << 32),
+                                               crash_size, alignment);
+               /*
+                * crashkernel=X reserve below 4G fails? Try MAXMEM
+                */
+               if (!high && !crash_base)
+                       crash_base = memblock_find_in_range(alignment,
+                                               CRASH_KERNEL_ADDR_HIGH_MAX,
+                                               crash_size, alignment);
 
Quoted 5 lines

arch/x86/kernel/setup.c static void __init reserve_crashkernel(void)

Quoted 3 lines
arch/x86/kernel/setup.c +0 -1
                if (!crash_base) {
                        pr_info("crashkernel reservation failed - No suitable area found.\n");
--

User should change crashkernel=X to crashkernel=X,high.

crashkernel=X is more straightforward. IMHO, It should be a more general crash kernel reservation parameter and it shouldn't reserve at low only.

Quoted 3 lines

As user could forget to update kexec-tools to utilize ",high" feature, and get kdump later fail later.

old kexec-tools should fail to load kernel high anyway, right?

Thanks
WANG Chao

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#3 lore raw
In reply to Yinghai Lu
Quoted 51 lines · Yinghai Lu
Quoted 50 lines · WANG Chao

Now crashkernel=X will fail out if there's not enough memory at low (below 896M). What makes sense for crashkernel=X would be:

- First try to reserve X below 896M (for being compatible with old kexec-tools).
- If fails, try to reserve X below 4G (swiotlb need to stay below 4G).
- If fails, try to reserve X from MAXMEM top down.

So that user can easily reserve large memory with crashkernel=X instead of crashkernel=X,high. It's more transparent and user-friendly.

If crashkernel is large and the reserved is beyond 896M, old kexec-tools won't be compatible with new kernel for most of time.

kexec will fail out immediately in this case. But the failure could be expected, because old kexec users should not try to reserve that large amount of memory at the first place.

On the other hand, old kexec also will fail on old kernel when there's not enough low memory to reserve a large crash kernel area. So the failure of old kexec is consistent between old kernel and new kernel.

---
 arch/x86/kernel/setup.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
arch/x86/kernel/setup.c +14 -0
index f0de629..38e6c1f 100644
@@ -593,6 +593,20 @@ static void __init reserve_crashkernel(void)
                                        high ? CRASH_KERNEL_ADDR_HIGH_MAX :
                                               CRASH_KERNEL_ADDR_LOW_MAX,
                                        crash_size, alignment);
+               /*
+                * crashkernel=X reserve below 896M fails? Try below 4G
+                */
+               if (!high && !crash_base)
+                       crash_base = memblock_find_in_range(alignment,
+                                               (1ULL << 32),
+                                               crash_size, alignment);
+               /*
+                * crashkernel=X reserve below 4G fails? Try MAXMEM
+                */
+               if (!high && !crash_base)
+                       crash_base = memblock_find_in_range(alignment,
+                                               CRASH_KERNEL_ADDR_HIGH_MAX,
+                                               crash_size, alignment);
 
Quoted 5 lines

arch/x86/kernel/setup.c static void __init reserve_crashkernel(void)

Quoted 3 lines
arch/x86/kernel/setup.c +0 -1
                if (!crash_base) {
                        pr_info("crashkernel reservation failed - No suitable area found.\n");
--

User should change crashkernel=X to crashkernel=X,high.

I think if we can extend old syntax of crashkernel=X, then it makes life really easy for users.

Quoted 3 lines

As user could forget to update kexec-tools to utilize ",high" feature, and get kdump later fail later.

Previously high reservation (reservation above 896M) will anyway fail. So instead of failing, if we try reservation in higher memory areas why that would break old kexec-tools?

IOW, previously anyway kexec-tools will not work as no memory will be reserved in higher memory area. Now memory will be reserved but old kexec-tools should fail as it can't load in that area.

If that works, then one would use crashkernel=X,high only if he is particualr that memory reservation comes from area above 4G (despite the fact that same memory could have been reserved below 4G too).

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#4 lore raw
Quoted 8 lines · Vivek Goyal
Quoted 2 lines · Yinghai Lu

On Mon, Oct 14, 2013 at 4:46 AM, WANG Chao <[email protected]> wrote:
User should change crashkernel=X to crashkernel=X,high.

I think if we can extend old syntax of crashkernel=X, then it makes life really easy for users.

Quoted 1 line
Quoted 6 lines
Quoted 2 lines

As user could forget to update kexec-tools to utilize ",high" feature, and get kdump later fail later.

Previously high reservation (reservation above 896M) will anyway fail. So instead of failing, if we try reservation in higher memory areas why that would break old kexec-tools?

If thel old kexec-tools would fail, we should let them fail early as possible, do not reserve at first point.

Quoted 8 lines

IOW, previously anyway kexec-tools will not work as no memory will be reserved in higher memory area. Now memory will be reserved but old kexec-tools should fail as it can't load in that area.

If that works, then one would use crashkernel=X,high only if he is particualr that memory reservation comes from area above 4G (despite the fact that same memory could have been reserved below 4G too).

My point:
Push user to use ",high" as more as possible, so we only to handle one path eventually.
for old kernel, leave them to use old grammer. do not need to change it.

Also boot loader should always have different entry for old kernel and new kernel.

Thanks

Yinghai

#5 lore raw

On Thu, Oct 17, 2013 at 08:50:07PM -0700, Yinghai Lu wrote:

[..]

Quoted 6 lines
Quoted 3 lines

Previously high reservation (reservation above 896M) will anyway fail. So instead of failing, if we try reservation in higher memory areas why that would break old kexec-tools?

If thel old kexec-tools would fail, we should let them fail early as possible, do not reserve at first point.

A user does not care if they get the message "memory nor reserved" or if they get the message that "could not find a suitable memory hole at address X".

Quoted 6 lines
Quoted 6 lines

IOW, previously anyway kexec-tools will not work as no memory will be reserved in higher memory area. Now memory will be reserved but old kexec-tools should fail as it can't load in that area.

Quoted 8 lines
Quoted 3 lines

If that works, then one would use crashkernel=X,high only if he is particualr that memory reservation comes from area above 4G (despite the fact that same memory could have been reserved below 4G too).

My point:
Push user to use ",high" as more as possible, so we only to handle one path eventually.
for old kernel, leave them to use old grammer. do not need to change it.

I don't understand this. Why we should push users to use ",high" syntax. That is an option. Those who want to use it, should use it.

We have been using crashkernel=XM for a long time now. And it makes sense to extend this option to be able to reserve memory at higher addresses if asked memory is not available at lower addresses.

You are not thinking about ease of use here for existing users.

Quoted 3 lines

Also boot loader should always have different entry for old kernel and new kernel.

What does memory reservation location has to do with kernel entry point? If it is a 64bit bzImage, we will use 64bit entry point by default, isn't it? Does not matter whether memory is reserved above 4G or not.

I think it makes sense that existing crashkernel=XM usres be able to reserve memory in higher memory area if sufficient memory is not available below 896M or below 4G. Those who always want memory reservation above 4G they should use ",high" syntax and enforce memory allocation above 4G.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#6 lore raw
Quoted 7 lines · Vivek Goyal

On Thu, Oct 17, 2013 at 08:50:07PM -0700, Yinghai Lu wrote:

[..]

Quoted 4 lines
Quoted 3 lines

Previously high reservation (reservation above 896M) will anyway fail. So instead of failing, if we try reservation in higher memory areas why that would break old kexec-tools?

Quoted 6 lines
Quoted 2 lines

If thel old kexec-tools would fail, we should let them fail early as possible, do not reserve at first point.

A user does not care if they get the message "memory nor reserved" or if they get the message that "could not find a suitable memory hole at address X".

you are asking for trouble.

Now we have two paths:
1. old kernel with old kexec tools. crashkernel=XM, work,
the new kernel with old kexec tools still working with crashkernel=XM
2. old kernel with old kernel tools, crashkernel=XM, not working. as X is too big.
then user update to new kernel AND new kexec-tools, and crashkernel=XM,high

with this patch, you will need to test new kernel all old kexec tools to make sure
it will fail later instead of fail early to remind them to update kexec tools. Also would make user to guess and try to make new kernel to work with old kexec-tools

Quoted 16 lines
Quoted 7 lines
Quoted 7 lines

IOW, previously anyway kexec-tools will not work as no memory will be reserved in higher memory area. Now memory will be reserved but old kexec-tools should fail as it can't load in that area.

Quoted 8 lines
Quoted 3 lines

If that works, then one would use crashkernel=X,high only if he is particualr that memory reservation comes from area above 4G (despite the fact that same memory could have been reserved below 4G too).

My point:
Push user to use ",high" as more as possible, so we only to handle one path eventually.
for old kernel, leave them to use old grammer. do not need to change it.

Quoted 8 lines

I don't understand this. Why we should push users to use ",high" syntax. That is an option. Those who want to use it, should use it.

We have been using crashkernel=XM for a long time now. And it makes sense to extend this option to be able to reserve memory at higher addresses if asked memory is not available at lower addresses.

You are not thinking about ease of use here for existing users.

most existing user don't need to do anything. just with new kernel and old kexec tools.

those system that did not work kexec before because XM is too big, they have to update kexec tools, and use ",high"

Make it simple, less error.

Quoted 5 lines
Quoted 4 lines

Also boot loader should always have different entry for old kernel and new kernel.

Quoted 8 lines

What does memory reservation location has to do with kernel entry point? If it is a 64bit bzImage, we will use 64bit entry point by default, isn't it? Does not matter whether memory is reserved above 4G or not.

I think it makes sense that existing crashkernel=XM usres be able to reserve memory in higher memory area if sufficient memory is not available below 896M or below 4G. Those who always want memory reservation above 4G they should use ",high" syntax and enforce memory allocation above 4G.

We already support above 4G, what is point for trying below 4G?

You could get more bug report about new kernel with old kexec-tools, as old kexec-tools could work with range between 896M and 4G in some case, but not all.

Thanks

Yinghai

#7 lore raw

On Fri, Oct 18, 2013 at 10:45:43PM -0700, Yinghai Lu wrote:

[..]

Quoted 13 lines
Quoted 3 lines

A user does not care if they get the message "memory nor reserved" or if they get the message that "could not find a suitable memory hole at address X".

you are asking for trouble.

Now we have two paths:
1. old kernel with old kexec tools. crashkernel=XM, work,
the new kernel with old kexec tools still working with crashkernel=XM
2. old kernel with old kernel tools, crashkernel=XM, not working. as X is too big.
then user update to new kernel AND new kexec-tools, and crashkernel=XM,high

Quoted 5 lines

with this patch, you will need to test new kernel all old kexec tools to make sure
it will fail later instead of fail early to remind them to update kexec tools. Also would make user to guess and try to make new kernel to work with old kexec-tools

Seriously, I don't understand what are you trying to say above. You will need to explain your concern more clearly.

IIUC, you are trying to say that with new kernel old kexec-tools will fail at a different failure point. I don't see why that is a problem. It still fails.

[..]

Quoted 9 lines
Quoted 1 line

You are not thinking about ease of use here for existing users.

most existing user don't need to do anything. just with new kernel and old kexec tools.

those system that did not work kexec before because XM is too big, they have to update kexec tools, and use ",high"

Make it simple, less error.

No, it is not that simple. Think from a distribution's perspective also. We have the logic to scale reserved memory based on physical memory present in the system. Now we are seeing bigger memory systems (which would not have worked in the past). We still want to retain the existing logic and not switch to crashkernel=x,high. One does not have to. It makes life simpler.

Same logic working both with smaller memory systems as well as large memory systems. One should not have to choose a different command line because there is more physical RAM present in the system.

Quoted 10 lines
Quoted 10 lines
Quoted 5 lines

Also boot loader should always have different entry for old kernel and new kernel.

What does memory reservation location has to do with kernel entry point? If it is a 64bit bzImage, we will use 64bit entry point by default, isn't it? Does not matter whether memory is reserved above 4G or not.

Quoted 6 lines
Quoted 4 lines

I think it makes sense that existing crashkernel=XM usres be able to reserve memory in higher memory area if sufficient memory is not available below 896M or below 4G. Those who always want memory reservation above 4G they should use ",high" syntax and enforce memory allocation above 4G.

We already support above 4G, what is point for trying below 4G?

Because it is not *required* to reserve memory above 4G. Because we want same command line to work with both small memory systems as well as large memory systems and we don't care whether memory is reserved below 4G or above 4G. What does matter though that we don't have to worry about switching command line option if it is large memory system.

Quoted 4 lines

You could get more bug report about new kernel with old kexec-tools, as old kexec-tools could work with range between 896M and 4G in some case, but not all.

So you are worried that if we lift the artificial cap, and old kexec-tools start working that will be a problem. I don't think that is an issue. People worry about something existing broken. It does not break existing kexec-tools. If in some cases they work with memory reserved at higher addresses, where is the problem. Even in the past one could do crashkernel=X@Y and if Y is above 896M and if kexec-tools works with that what's wrong. I have not seen people complaining about it.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#8 lore raw
Quoted 6 lines · Vivek Goyal

On Fri, Oct 18, 2013 at 10:45:43PM -0700, Yinghai Lu wrote:

IIUC, you are trying to say that with new kernel old kexec-tools will fail at a different failure point. I don't see why that is a problem. It still fails.

Yes, that could cause confusion.

We already knew it would fail possible at most later, we should make it skip allocation during first kernel booting.

Quoted 10 lines

[..]

Quoted 8 lines
Quoted 1 line

You are not thinking about ease of use here for existing users.

most existing user don't need to do anything. just with new kernel and old kexec tools.

those system that did not work kexec before because XM is too big, they have to update kexec tools, and use ",high"

Quoted 8 lines
Quoted 1 line

Make it simple, less error.

No, it is not that simple. Think from a distribution's perspective also. We have the logic to scale reserved memory based on physical memory present in the system. Now we are seeing bigger memory systems (which would not have worked in the past). We still want to retain the existing logic and not switch to crashkernel=x,high. One does not have to. It makes life simpler.

distribution should go with ",high" for 64 bit kernel and new kexec-tools. for 32bit kernel, you still can have ",high" or not, as ",high" is ignored.

Quoted 4 lines

Same logic working both with smaller memory systems as well as large memory systems. One should not have to choose a different command line because there is more physical RAM present in the system.

",high" is working even on smaller memory sytem.

Quoted 9 lines
Quoted 3 lines

We already support above 4G, what is point for trying below 4G?

Because it is not *required* to reserve memory above 4G. Because we want same command line to work with both small memory systems as well as large memory systems and we don't care whether memory is reserved below 4G or above 4G. What does matter though that we don't have to worry about switching command line option if it is large memory system.

",high" will work smaller or large memory system after you install new kexec tools.

Again, for distribution, when new kernel is added, new kernel will all have ",high"
and new kexec-tools get installed.

Even we want to extend crashkernel=XM, then i would like to have it identical to crashkernel=XM,high instead.

Thanks

Yinghai

author
#9 lore raw
Quoted 4 lines · Yinghai Lu
Quoted 3 lines · Vivek Goyal

On Fri, Oct 18, 2013 at 10:45:43PM -0700, Yinghai Lu wrote:

Quoted 8 lines
Quoted 3 lines

IIUC, you are trying to say that with new kernel old kexec-tools will fail at a different failure point. I don't see why that is a problem. It still fails.

Yes, that could cause confusion.

We already knew it would fail possible at most later, we should make it skip allocation during first kernel booting.

It's said:

crashkernel=size[KMG][@offset[KMG]]
[KNL] Using kexec, Linux can switch to a 'crash kernel' upon panic. This parameter reserves the physical memory region [offset, offset + size] for that kernel image. If '@offset' is omitted, then a suitable offset is selected automatically. Check Documentation/kdump/kdump.txt for further details.

Nothing mentions that crashkernel=X is restricted to reserve at low.

It could be more confusing that crashkernel=X (say 1G) can not find a suitable area while actually there's enough memory out there.

Current situation is the combination of crashkernel X/high/low is quite complicated, it's more like supposed to be used by the pros.

Truth is, most user, after upgrading to new kernel, would want reserve large by just changing to crashkernel=1G, not be forced to use a new format craskernel=1G,high.

I think crashkernel=XM,high is really supposed to be used when user indeed want to reserve from high.

Like Vivek said, failing at different point shouldn't be a problem. That's an incorrect configuration. When crashkernel=1G,high, old kexec-tools still fails the same way. That could cause confusion, in your word.

Let me put it in an example, a user want to utilize this new kernel feature to reserve 1G for crash kernel but not upgrade kexec-tools,

- W/o this patch:
First he would try crashkernel=1G, but failed to reserve. Second time, he goes with crashkernel=1G,high, reservation is fine but kexec fails to load. Upgrading kexec-tools is essential to him.

- W/ this patch:
First he would try crashkernel=1G, reservation is ok but kexec fails to load the same way as the case of crashkernel=1G,high. Upgrading kexec-tools is essential to him.

The point is old kexec-tools can't load high, no matter by what kind of crashkernel cmdline to reserve at high.

Quoted 20 lines
Quoted 11 lines

[..]

Quoted 8 lines
Quoted 1 line

You are not thinking about ease of use here for existing users.

most existing user don't need to do anything. just with new kernel and old kexec tools.

those system that did not work kexec before because XM is too big, they have to update kexec tools, and use ",high"

Quoted 8 lines
Quoted 1 line

Make it simple, less error.

No, it is not that simple. Think from a distribution's perspective also. We have the logic to scale reserved memory based on physical memory present in the system. Now we are seeing bigger memory systems (which would not have worked in the past). We still want to retain the existing logic and not switch to crashkernel=x,high. One does not have to. It makes life simpler.

Quoted 2 lines

distribution should go with ",high" for 64 bit kernel and new kexec-tools. for 32bit kernel, you still can have ",high" or not, as ",high" is ignored.

Why ",high" is becoming a default one instead of "XM"?

Isn't that the kernel's job to find a most suitable area at best guess in this case? By this case, I mean kernel have no idea kexec-tools is new or old.

Quoted 8 lines
Quoted 6 lines

Same logic working both with smaller memory systems as well as large memory systems. One should not have to choose a different command line because there is more physical RAM present in the system.

",high" is working even on smaller memory sytem.

Yeah, you're right.

Quoted 14 lines
Quoted 10 lines
Quoted 4 lines

We already support above 4G, what is point for trying below 4G?

Because it is not *required* to reserve memory above 4G. Because we want same command line to work with both small memory systems as well as large memory systems and we don't care whether memory is reserved below 4G or above 4G. What does matter though that we don't have to worry about switching command line option if it is large memory system.

",high" will work smaller or large memory system after you install new kexec tools.

Quoted 6 lines

Again, for distribution, when new kernel is added, new kernel will all have ",high"
and new kexec-tools get installed.

Even we want to extend crashkernel=XM, then i would like to have it identical to crashkernel=XM,high instead.

Why would you like crashkernel=XM is identical to crashkernel=XM,high?

That's not an option if you want new kernel to be compatible with old kexec-tools.

Thanks
WANG Chao

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#10 lore raw
Quoted 3 lines · WANG Chao

I think crashkernel=XM,high is really supposed to be used when user indeed want to reserve from high.

No. Keep the all 64bit to stay high, make thing simple. instead of some low and some use high.

Quoted 5 lines

Like Vivek said, failing at different point shouldn't be a problem. That's an incorrect configuration. When crashkernel=1G,high, old kexec-tools still fails the same way. That could cause confusion, in your word.

If it would fail later, we should let it fail early as possible.

Quoted 9 lines

Let me put it in an example, a user want to utilize this new kernel feature to reserve 1G for crash kernel but not upgrade kexec-tools,

- W/o this patch:
First he would try crashkernel=1G, but failed to reserve. Second time, he goes with crashkernel=1G,high, reservation is fine but kexec fails to load. Upgrading kexec-tools is essential to him.

Quoted 7 lines

- W/ this patch:
First he would try crashkernel=1G, reservation is ok but kexec fails to load the same way as the case of crashkernel=1G,high. Upgrading kexec-tools is essential to him.

The point is old kexec-tools can't load high, no matter by what kind of crashkernel cmdline to reserve at high.

old kexec-tools could work cross 892M in some case.
That will confuse the user, as it works some time on some setup, but does not work on other setup.

Thanks

Yinghai

#11 lore raw

On Thu, Oct 24, 2013 at 12:13:41PM -0700, Yinghai Lu wrote:

[..]

Quoted 2 lines

old kexec-tools could work cross 892M in some case. That will confuse the user,

If it could work beyong 892MB, then why are you limiting user to 896MB only. That sounds wrong.

Quoted 2 lines

as it works some time on some setup, but does
not work on other setup.

If it can work above 896MB, then those other setup should be fixed.

All the existing users of crashkernel=xM *will not* be affected by this change. Memory reservation will still be below 892MB.

Memory reservation will gove above 896MB, only if user chooses to reserve more memory. If kexec works in that case, its a win-win situation. If it does not and fails, then it is status quo is maintained.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#12 lore raw
In reply to Yinghai Lu
Quoted 4 lines · Yinghai Lu
Quoted 3 lines · Vivek Goyal

On Fri, Oct 18, 2013 at 10:45:43PM -0700, Yinghai Lu wrote:

Quoted 8 lines
Quoted 3 lines

IIUC, you are trying to say that with new kernel old kexec-tools will fail at a different failure point. I don't see why that is a problem. It still fails.

Yes, that could cause confusion.

We already knew it would fail possible at most later, we should make it skip allocation during first kernel booting.

One could upgrade kexec-tools later. So Kernel does not know whether user space is able to handle higher memory regions or not. So forcing kernel to skip memory allocation does not sound right to me.

Also even with memory reserved high, old kexec-tools which can't deal with it will fail anyway. So I don't agree that it is a problem.

[..]

Quoted 9 lines
Quoted 6 lines

No, it is not that simple. Think from a distribution's perspective also. We have the logic to scale reserved memory based on physical memory present in the system. Now we are seeing bigger memory systems (which would not have worked in the past). We still want to retain the existing logic and not switch to crashkernel=x,high. One does not have to. It makes life simpler.

distribution should go with ",high" for 64 bit kernel and new kexec-tools. for 32bit kernel, you still can have ",high" or not, as ",high" is ignored.

Just because we have the facility to allocate memory starting from top, does not mean that we should kill other options and force user to use this option.

With crashkernel=X,high, a low memory will be reserved for swiotlb. And I don't want that extra memory reservation. I am more than happy to reserve memory below 4G and not do low memory reservation.

[..]

Quoted 2 lines

Even we want to extend crashkernel=XM, then i would like to have it identical to crashkernel=XM,high instead.

You are assuming that crashkernel=xM,high is always good. I am arguing that because it requires low memory reservation for swiotlb, when IOMMU is not around, it will end up reserving more memory. So it is not necessarily better than reserving memory below 4G.

Hence both crashkernel=xM and crashkernel=XM,high have their own usage. We have been using crashkernel=xM and we know it works. So extending it to be able to allocate memory from higher regions, if sufficient memory is not available in lower regions makes sense. Memory reservation below 4G is more efficient due to not requiring swiotlb. And crashkernel=xM has been working for us and users are familiar with it.

So I don't see a point that why would you try to block any move to extend crashkernel=xM semantics.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#13 lore raw
Quoted 11 lines · Vivek Goyal

On Wed, Oct 23, 2013 at 11:11:51PM -0700, Yinghai Lu wrote:

Hence both crashkernel=xM and crashkernel=XM,high have their own usage. We have been using crashkernel=xM and we know it works. So extending it to be able to allocate memory from higher regions, if sufficient memory is not available in lower regions makes sense. Memory reservation below 4G is more efficient due to not requiring swiotlb. And crashkernel=xM has been working for us and users are familiar with it.

So I don't see a point that why would you try to block any move to extend crashkernel=xM semantics.

Make the thing simple.
Keep them separately, leave crashkernel=xM to old kexec-tools mostly and keep crashkernel=xM,high to newer kexec-tools as needed.

Thanks

Yinghai

#14 lore raw
Quoted 10 lines · Yinghai Lu
Quoted 9 lines · Vivek Goyal

On Wed, Oct 23, 2013 at 11:11:51PM -0700, Yinghai Lu wrote:

Hence both crashkernel=xM and crashkernel=XM,high have their own usage. We have been using crashkernel=xM and we know it works. So extending it to be able to allocate memory from higher regions, if sufficient memory is not available in lower regions makes sense. Memory reservation below 4G is more efficient due to not requiring swiotlb. And crashkernel=xM has been working for us and users are familiar with it.

Quoted 6 lines
Quoted 2 lines

So I don't see a point that why would you try to block any move to extend crashkernel=xM semantics.

Make the thing simple.
Keep them separately, leave crashkernel=xM to old kexec-tools mostly and keep crashkernel=xM,high to newer kexec-tools as needed.

I am keeping things simple by making sure that both old kexec-tools and new kexec-tools can use crashkernel=xM and one does not have to choose between two based on what kexec-tools version you are using.

Also keeping things simple by not trying to *impose* a new crashkernel= syntax on existing crashkernel=xM users.

Hence extending the semantics of crashkernel=xM makes sense to me.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#15 lore raw
Quoted 4 lines · Vivek Goyal

On Thu, Oct 24, 2013 at 12:15:25PM -0700, Yinghai Lu wrote:

Also keeping things simple by not trying to *impose* a new crashkernel= syntax on existing crashkernel=xM users.

Existing user that have crashkernel=xM working with their old kernel and old kexec-tools, they still could keep their old command line and old kexec-tools
with new updated kernel.
We should not change semantics to surprise them.

Thanks

Yinghai

#16 lore raw
Quoted 11 lines · Yinghai Lu
Quoted 4 lines · Vivek Goyal

On Thu, Oct 24, 2013 at 12:15:25PM -0700, Yinghai Lu wrote:

Also keeping things simple by not trying to *impose* a new crashkernel= syntax on existing crashkernel=xM users.

Existing user that have crashkernel=xM working with their old kernel and old kexec-tools, they still could keep their old command line and old kexec-tools
with new updated kernel.
We should not change semantics to surprise them.

Old users will get reservation still below 896MB.

It will go above 896MB only if memory could not be allocated below 896MB.

In the past reservation will fail and kexec-tools will fail. Now reservation will succeed but kexec-tools will fail.

So end result a user sees is that kexec-tools fails. So I don't see how we are breaking existing installations or user setups.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#17 lore raw
Quoted 13 lines · Vivek Goyal
Quoted 11 lines · Yinghai Lu
Quoted 4 lines · Vivek Goyal

On Thu, Oct 24, 2013 at 12:15:25PM -0700, Yinghai Lu wrote:

Also keeping things simple by not trying to *impose* a new crashkernel= syntax on existing crashkernel=xM users.

Existing user that have crashkernel=xM working with their old kernel and old kexec-tools, they still could keep their old command line and old kexec-tools
with new updated kernel.
We should not change semantics to surprise them.

Quoted 6 lines

Old users will get reservation still below 896MB.

It will go above 896MB only if memory could not be allocated below 896MB.

In the past reservation will fail and kexec-tools will fail. Now reservation will succeed but kexec-tools will fail.

Only corner case I can see is where memory is reserved higher, old kexec-tools is not smart enough to figure out in advance that kernel will not boot at this high address and still goes ahead and loads the kernel and crash dump fails at run time.

But this will happen only if memory reservation below 896MB fails. And in most of the cases after kernel upgrade, memory should still come from memory region below 896MB.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#18 lore raw
In reply to Vivek Goyal
Quoted 15 lines · Vivek Goyal
Quoted 11 lines · Yinghai Lu
Quoted 4 lines · Vivek Goyal

On Thu, Oct 24, 2013 at 12:15:25PM -0700, Yinghai Lu wrote:

Also keeping things simple by not trying to *impose* a new crashkernel= syntax on existing crashkernel=xM users.

Existing user that have crashkernel=xM working with their old kernel and old kexec-tools, they still could keep their old command line and old kexec-tools
with new updated kernel.
We should not change semantics to surprise them.

Old users will get reservation still below 896MB.

Quoted 7 lines

It will go above 896MB only if memory could not be allocated below 896MB.

In the past reservation will fail and kexec-tools will fail. Now reservation will succeed but kexec-tools will fail.

So end result a user sees is that kexec-tools fails. So I don't see how we are breaking existing installations or user setups.

case could be: if user add more memory and put more pcie cards, and second kernel will need more ram and OOM there.
so user could just increase crashkernel=512M to crashkernel=1G.

without Cong's patch, kernel will fail to reserve, and user would dig to change it
to crashkernel=1G,high, and update kexec-tools.

with Cong's patch, kernel will reserve other range like between 896 and 4G, old kexec-tools either
fail to load second kernel or hang in purgatory or early stage of second kernel, or other unknown behavior.

I would think first path is much clear and predicted.

If my memory is right, HPA did not like idea that we try below 896M, and then under 4G and then above 4G.
and want us to have ",high" solution. Not sure if he would change his mind. Also you will need and #ifdef CONFIG_X86_64 for 896M/4G searching code so it will not confuse the
32bit arch.

Thanks

Yinghai

#21 lore raw
In reply to Yinghai Lu
Quoted 18 lines · Yinghai Lu
Quoted 17 lines · Vivek Goyal
Quoted 11 lines · Yinghai Lu
Quoted 4 lines · Vivek Goyal

On Thu, Oct 24, 2013 at 12:15:25PM -0700, Yinghai Lu wrote:

Also keeping things simple by not trying to *impose* a new crashkernel= syntax on existing crashkernel=xM users.

Existing user that have crashkernel=xM working with their old kernel and old kexec-tools, they still could keep their old command line and old kexec-tools
with new updated kernel.
We should not change semantics to surprise them.

Old users will get reservation still below 896MB.

It will go above 896MB only if memory could not be allocated below 896MB.

Quoted 8 lines
Quoted 5 lines

In the past reservation will fail and kexec-tools will fail. Now reservation will succeed but kexec-tools will fail.

So end result a user sees is that kexec-tools fails. So I don't see how we are breaking existing installations or user setups.

case could be: if user add more memory and put more pcie cards, and second kernel will need more ram and OOM there.

Now makedumpfile supports cyclic mode by default. So one does not have to necessarily linearly scale reserved memory based on physical memory present in the box.

Quoted 1 line

so user could just increase crashkernel=512M to crashkernel=1G.

If user has new makedumpfile, OOM should not happen and one should not have to increase memory reservation.

Quoted 9 lines

without Cong's patch, kernel will fail to reserve, and user would dig to change it
to crashkernel=1G,high, and update kexec-tools.

with Cong's patch, kernel will reserve other range like between 896 and 4G, old kexec-tools either
fail to load second kernel or hang in purgatory or early stage of second kernel, or other unknown behavior.

I understand your concern about memory being reserved high and kexec just hanging. Only thing I am arguing is that the number of cases where it will happen is small.

- First of all for all old existing kexec-tools memory will stil come from 896MB.
- Old kexec-tools enforced that purgatory is loaded below 2G. So if memory is reserved above 2G, kexec-tools will fail.

So only problematic case seems to be that if user increased crashkernel=X value and memory got reserved between 896M and 2G. But this is not same as breaking old setups as old setups anyway never worked with this configuration.

You argument that user will research and upgrade kexec-tools and use crashkernel=X,high, then it holds true for the case where memory is reserved between 896M and 2G.

I personally think that it is easier for a user to not change any kernel parameters with kernel and kexec-tools upgrade and still be able to work with large memory systems. So the benefit of extending the semantics of existing parameter seems to be outweighing the downside of side, IMHO.

Quoted 5 lines

I would think first path is much clear and predicted.

If my memory is right, HPA did not like idea that we try below 896M, and then under 4G and then above 4G.

hpa, I know you did not like the idea in the past. Is it still the case.

IMHO, I like the fact that existing users still be able to work with crashkernel=X and not forced to switch to crashkernel=x,high and also incur the penalty of reserving extra memory for software iotlb.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

author
#22 lore raw
Quoted 72 lines · Vivek Goyal
Quoted 18 lines · Yinghai Lu
Quoted 17 lines · Vivek Goyal
Quoted 11 lines · Yinghai Lu
Quoted 4 lines · Vivek Goyal

On Thu, Oct 24, 2013 at 12:15:25PM -0700, Yinghai Lu wrote:

Also keeping things simple by not trying to *impose* a new crashkernel= syntax on existing crashkernel=xM users.

Existing user that have crashkernel=xM working with their old kernel and old kexec-tools, they still could keep their old command line and old kexec-tools
with new updated kernel.
We should not change semantics to surprise them.

Old users will get reservation still below 896MB.

It will go above 896MB only if memory could not be allocated below 896MB.

Quoted 8 lines
Quoted 5 lines

In the past reservation will fail and kexec-tools will fail. Now reservation will succeed but kexec-tools will fail.

So end result a user sees is that kexec-tools fails. So I don't see how we are breaking existing installations or user setups.

case could be: if user add more memory and put more pcie cards, and second kernel will need more ram and OOM there.

Now makedumpfile supports cyclic mode by default. So one does not have to necessarily linearly scale reserved memory based on physical memory present in the box.

Quoted 1 line

so user could just increase crashkernel=512M to crashkernel=1G.

If user has new makedumpfile, OOM should not happen and one should not have to increase memory reservation.

Quoted 9 lines

without Cong's patch, kernel will fail to reserve, and user would dig to change it
to crashkernel=1G,high, and update kexec-tools.

with Cong's patch, kernel will reserve other range like between 896 and 4G, old kexec-tools either
fail to load second kernel or hang in purgatory or early stage of second kernel, or other unknown behavior.

I understand your concern about memory being reserved high and kexec just hanging. Only thing I am arguing is that the number of cases where it will happen is small.

- First of all for all old existing kexec-tools memory will stil come from 896MB.
- Old kexec-tools enforced that purgatory is loaded below 2G. So if memory is reserved above 2G, kexec-tools will fail.

So only problematic case seems to be that if user increased crashkernel=X value and memory got reserved between 896M and 2G. But this is not same as breaking old setups as old setups anyway never worked with this configuration.

You argument that user will research and upgrade kexec-tools and use crashkernel=X,high, then it holds true for the case where memory is reserved between 896M and 2G.

I personally think that it is easier for a user to not change any kernel parameters with kernel and kexec-tools upgrade and still be able to work with large memory systems. So the benefit of extending the semantics of existing parameter seems to be outweighing the downside of side, IMHO.

Quoted 1 line
Quoted 6 lines
Quoted 4 lines

I would think first path is much clear and predicted.

If my memory is right, HPA did not like idea that we try below 896M, and then under 4G and then above 4G.

hpa, I know you did not like the idea in the past. Is it still the case.

Hi, hpa

Do you still not like this idea? Could you make it more clear to us please?

Thanks
Chao

Quoted 7 lines

IMHO, I like the fact that existing users still be able to work with crashkernel=X and not forced to switch to crashkernel=x,high and also incur the penalty of reserving extra memory for software iotlb.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#23 lore raw
In reply to Yinghai Lu
Quoted 6 lines

Again, for distribution, when new kernel is added, new kernel will all have ",high"
and new kexec-tools get installed.

Even we want to extend crashkernel=XM, then i would like to have it identical to crashkernel=XM,high instead.

My points: I can accept the approach of extending crashkernel=XM to use high mem firstly if we could remove the extra 72M in low mem, but apparently it's not doable now. So I would vote for Chao's patch.

Also newer kernel use more complex ",high" then older kernel looks not good.

Thanks
Dave

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#24 lore raw
In reply to WANG Chao
Quoted 13 lines · WANG Chao

Now crashkernel=X will fail out if there's not enough memory at low (below 896M). What makes sense for crashkernel=X would be:

- First try to reserve X below 896M (for being compatible with old kexec-tools).
- If fails, try to reserve X below 4G (swiotlb need to stay below 4G).
- If fails, try to reserve X from MAXMEM top down.

arch/x86/kernel/setup.c +0 -0
index f0de629..38e6c1f 100644

arch/x86/kernel/setup.c static void __init reserve_crashkernel(void) · line 602

Quoted 10 lines
arch/x86/kernel/setup.c +7 -0
@@ -593,6 +593,20 @@ static void __init reserve_crashkernel(void)
                                        high ? CRASH_KERNEL_ADDR_HIGH_MAX :
                                               CRASH_KERNEL_ADDR_LOW_MAX,
                                        crash_size, alignment);
+               /*
+                * crashkernel=X reserve below 896M fails? Try below 4G
+                */
+               if (!high && !crash_base)
+                       crash_base = memblock_find_in_range(alignment,
+                                               (1ULL << 32),
+                                               crash_size, alignment);

Another problem, it would allocate range in [0,4g) for 32bit, if the user have crashkernel=512M or plus.

arch/x86/kernel/setup.c static void __init reserve_crashkernel(void) · line 612

Quoted 10 lines
arch/x86/kernel/setup.c +7 -0
+               /*
+                * crashkernel=X reserve below 4G fails? Try MAXMEM
+                */
+               if (!high && !crash_base)
+                       crash_base = memblock_find_in_range(alignment,
+                                               CRASH_KERNEL_ADDR_HIGH_MAX,
+                                               crash_size, alignment);
 
                if (!crash_base) {
                        pr_info("crashkernel reservation failed - No suitable area found.\n");

Yinghai

author
#25 lore raw
Quoted 19 lines · Yinghai Lu
Quoted 18 lines · WANG Chao

Now crashkernel=X will fail out if there's not enough memory at low (below 896M). What makes sense for crashkernel=X would be:

- First try to reserve X below 896M (for being compatible with old kexec-tools).
- If fails, try to reserve X below 4G (swiotlb need to stay below 4G).
- If fails, try to reserve X from MAXMEM top down.

arch/x86/kernel/setup.c +2 -0
index f0de629..38e6c1f 100644
@@ -593,6 +593,20 @@ static void __init reserve_crashkernel(void)
                                        high ? CRASH_KERNEL_ADDR_HIGH_MAX :
                                               CRASH_KERNEL_ADDR_LOW_MAX,
                                        crash_size, alignment);
+               /*
+                * crashkernel=X reserve below 896M fails? Try below 4G
Quoted 8 lines

arch/x86/kernel/setup.c static void __init reserve_crashkernel(void) · line 602

Quoted 5 lines
arch/x86/kernel/setup.c +5 -0
+                */
+               if (!high && !crash_base)
+                       crash_base = memblock_find_in_range(alignment,
+                                               (1ULL << 32),
+                                               crash_size, alignment);

Another problem, it would allocate range in [0,4g) for 32bit, if the user have crashkernel=512M or plus.

Yes, should use #ifdef CONFIG_X86_64 .. #endif.
Thanks for pointing this out.

WANG Chao

Quoted 9 lines
Quoted 9 lines
arch/x86/kernel/setup.c +7 -0
+               /*
+                * crashkernel=X reserve below 4G fails? Try MAXMEM
+                */
+               if (!high && !crash_base)
+                       crash_base = memblock_find_in_range(alignment,
+                                               CRASH_KERNEL_ADDR_HIGH_MAX,
+                                               crash_size, alignment);
 
Quoted 5 lines

arch/x86/kernel/setup.c static void __init reserve_crashkernel(void) · line 612

Quoted 2 lines
arch/x86/kernel/setup.c +0 -0
                if (!crash_base) {
                        pr_info("crashkernel reservation failed - No suitable area found.\n");

Yinghai

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec